Apparatus and method for dynamic range transforming of images

ABSTRACT

An image processing apparatus comprises a receiver ( 201 ) for receiving an image signal which comprises at least an encoded image and a target display reference. The target display reference is indicative of a dynamic range of a target display for which the encoded image is encoded. A dynamic range processor ( 203 ) generates an output image by applying a dynamic range transform to the encoded image in response to the target display reference. An output ( 205 ) then outputs an output image signal comprising the output image, e.g. to a suitable display. The dynamic range transform may furthermore be performed in response to a display dynamic range indication received from a display. The invention may be used to generate an improved High Dynamic Range (HDR) image from e.g. a Low Dynamic Range (LDR) image, or vice versa.

CROSS-REFERENCE TO PRIOR APPLICATIONS

This application is a continuation of U.S. Ser. No. 14/346,765, filed onMar. 24, 2014, which is the U.S. National Phase application under 35U.S.C. § 371 of International Application No. PCT/IB2012/054984, filedon Sep. 20, 2012, which claims the benefit of U.S. ProvisionalApplication No. 61/588,731, filed on Jan. 20, 2012, European PatentApplication No. 12160557.0, filed on Mar. 21, 2012 and European PatentApplication No. 11182922.2, filed on Sep. 27, 2011. These applicationsare hereby incorporated by reference herein.

FIELD OF THE INVENTION

The invention relates to dynamic range transforms for images, and inparticular, but not exclusively to image processing to generate HighDynamic Range images from Low Dynamic Range images or to generate LowDynamic Range images from High Dynamic Range images.

BACKGROUND OF THE INVENTION

Digital encoding of various source signals has become increasinglyimportant over the last decades as digital signal representation andcommunication increasingly has replaced analogue representation andcommunication. Continuous research and development is ongoing in how toimprove the quality that can be obtained from encoded images and videosequences while at the same time keeping the data rate to acceptablelevels.

An important factor for perceived image quality is the dynamic rangethat can be reproduced when an image is displayed. Conventionally, thedynamic range of reproduced images has tended to be substantiallyreduced in relation to normal vision. Indeed, luminance levelsencountered in the real world span a dynamic range as large as 14 ordersof magnitude, varying from a moonless night to staring directly into thesun. Instantaneous luminance dynamic range and the corresponding humanvisual system response can fall between 10.000:1 and 100.000:1 on sunnydays or at night (bright reflections versus dark shadow regions).Traditionally, dynamic range of displays has been confined to about 2-3orders of magnitude, and also sensors had a limited range, e.g.<10.000:1depending on noise acceptability. Consequently, it hastraditionally been possible to store and transmit images in 8-bitgamma-encoded formats without introducing perceptually noticeableartifacts on traditional rendering devices. However, in an effort torecord more precise and livelier imagery, novel High Dynamic Range (HDR)image sensors that are capable of recording dynamic ranges of more than6 orders of magnitude have been developed. Moreover, most specialeffects, computer graphics enhancement and other post-production workare already routinely conducted at higher bit depths and with higherdynamic ranges.

Furthermore, the contrast and peak luminance of state-of-the-art displaysystems continues to increase. Recently, new prototype displays havebeen presented with a peak luminance as high as 3000 Cd/m² and contrastratios of 5-6 orders of magnitude (display native, the viewingenvironment will also affect the finally rendered contrast ratio, whichmay for daytime television viewing even drop below 50:1). It is expectedthat future displays will be able to provide even higher dynamic rangesand specifically higher peak luminances and contrast ratios. Whentraditionally encoded 8-bit signals are displayed on such displays,annoying quantization and clipping artifacts may appear. Moreover,traditional video formats offer insufficient headroom and accuracy toconvey the rich information contained in new HDR imagery.

As a result, there is a growing need for new approaches that allow aconsumer to fully benefit from the capabilities of state-of-the-art (andfuture) sensors and display systems. Preferably, representations of suchadditional information are backwards-compatible such that legacyequipment can still receive ordinary video streams, while newHDR-enabled devices can take full advantage of the additionalinformation conveyed by the new format. Thus, it is desirable thatencoded video data not only represents HDR images but also allowsencoding of the corresponding traditional Low Dynamic Range (LDR) imagesthat can be displayed on conventional equipment.

In order to successfully introduce HDR systems and to fully exploit thepromise of HDR, it is important that the approach taken provides bothbackwards compatibility and allows optimization or at least adaptationto HDR displays. However, this inherently involves a conflict betweenoptimization for HDR and optimization for traditional LDR.

For example, typically image content, such as video clips, will beprocessed in the studio (color grading & tone mapping) for optimalappearance on a specific display. Traditionally, such optimization hasbeen performed for LDR displays. For example, during production for astandard LDR display, color grading experts will balance many picturequality aspects to create the desired ‘look’ for the storyline. This mayinvolve balancing regional and local contrasts, sometimes evendeliberately clipping pixels. For example, on a display with relativelylow peak brightness, explosions or bright highlights are often severelyclipped to convey an impression of high brightness to the viewer (thesame thing happens for dark shadow details on displays with poor blacklevels). This operation will typically be performed assuming a nominalLDR display and traditionally displays have deviated relatively littlefrom such nominal LDR displays as indeed virtually all consumer displaysare LDR displays.

However, if the movie was adapted for an HDR target display, the outcomewould be very different. Indeed, the color experts would perform anoptimization that would result in a very different code mapping. Forexample, not only can highlights and shadow details be better preservedon HDR displays but these may also be optimized to have differentdistribution over mid-grey tones. Thus, an optimal HDR image is notachieved by a simple scaling of an LDR image by a value corresponding tothe difference in the white point luminances (the maximum achievablebrightness). Ideally, separate color gradings and tone mappings would beperformed for each possible dynamic range of a display. For example, onevideo sequence would be for a maximum white point luminance of 500Cd/m², one for 1000 Cd/m², one for 1500 Cd/m² etc. up to the maximumpossible brightness. A given display could then simply select the videosequence corresponding to its brightness. However, such an approach isimpractical as it requires a large number of video sequences to begenerated thereby increasing the resource required to generate thesedifferent video sequences. Furthermore, the storage and distributioncapacity required would increase substantially. Also, the approach wouldlimit the possible maximum display brightness level to discrete levelsthereby providing suboptimal performance for displays with maximumdisplay brightness levels in between the levels for which videosequences are being provided. Furthermore, such an approach will notallow future displays developed with higher maximum brightness levelsthan for the highest brightness level video sequence to be exploited.

Accordingly, it is expected that only a limited number of videosequences will be created at the content provision side, and it isexpected that automatic dynamic range conversions will be applied atlater points in the distribution chain to such video sequences in orderto generate a video sequence suitable for the specific display on whichthe video sequence is rendered. However, in such approaches theresulting image quality is highly dependent on the automatic dynamicrange conversion.

Hence, an improved approach for supporting different dynamic ranges forimages, and preferably for supporting different dynamic range images,would be advantageous.

SUMMARY OF THE INVENTION

Accordingly, the Invention seeks to preferably mitigate, alleviate oreliminate one or more of the above mentioned disadvantages singly or inany combination.

According to an aspect of the invention there is provided an imageprocessing apparatus comprising: a receiver for receiving an imagesignal, the image signal comprising at least a first encoded image and afirst target display reference, the first target display reference beingindicative of a dynamic range of a first target display for which thefirst encoded image is encoded; a dynamic range processor arranged togenerate an output image by applying a dynamic range transform to thefirst encoded image in response to the first target display reference;and an output for outputting an output image signal comprising theoutput image.

The invention may allow a system to support different dynamic rangeimages and/or displays. In particular, the approach may allow improveddynamic range transforms that can adapt to the specific characteristicsof the rendering of the image. In many scenarios an improved dynamicrange transform from LDR to HDR images or from HDR to LDR can beachieved.

In some embodiments, the dynamic range transform increases a dynamicrange of the output video signal relative to the first encoded image. Insome embodiments, the dynamic range transform decreases a dynamic rangeof the output video signal relative to the first encoded image.

A dynamic range corresponds to a rendering luminance range, i.e. to arange from a minimum light output to a maximum light output for therendered image. Thus, a dynamic range is not merely a ratio between amaximum value and a minimum value, or a quantization measure (such as anumber of bits), but corresponds to an actual luminance range for arendering of an image. Thus, a dynamic range may be a range of luminancevalues, e.g. measured in candela per square meter (cd/m²) which is alsoreferred to as nits. A dynamic range is thus the luminance range fromthe light output (brightness) corresponding to the lowest luminancevalue (often assumed to be absolute black i.e. no light output) to thelight output (brightness) corresponding to the highest luminance value.The dynamic range may specifically be characterized by the highest lightoutput value, also referred to as the white point, white pointluminance, white luminance or maximum luminance. For LDR images and LDRdisplays, the white point is typically 500 nits or less.

The output image signal may specifically be fed to a display having aspecific dynamic range, and thus the dynamic range transform may convertthe encoded image from a dynamic range indicated by the target displayreference to a dynamic range of the display on which the image isrendered.

The image may be an image of a moving image sequence, such as e.g. aframe or image of a video sequence. As another example, the image may bea permanent background or e.g. an overlay image such as graphics etc.

The first encoded image may specifically be an LDR image and the outputimage may be an HDR image. The first encoded image may specifically bean HDR image and the output image may be an LDR image.

In accordance with an optional feature of the invention, the firsttarget display reference comprises a white point luminance of the firsttarget display.

This may provide advantageous operation in a many embodiments. Inparticular, it may allow low complexity and/or low overhead whileproviding sufficient information to allow an improved dynamic rangetransform to be performed.

In accordance with an optional feature of the invention, the firsttarget display reference comprises an Electro Optical Transfer Functionindication for the first target display.

This may provide advantageous operation in a many embodiments. Inparticular, it may allow low complexity and/or low overhead whileproviding sufficient information to allow an improved dynamic rangetransform to be performed. The approach may in particular allow thedynamic range transform to also adapt to specific characteristics fore.g. midrange luminances. For example, it may allow the dynamic rangetransform to take into account differences in the gamma of the targetdisplay and the end-user display.

In accordance with an optional feature of the invention, the firsttarget display reference comprises a tone mapping indicationrepresenting a tone mapping used to generate the first encoded image forthe first target display.

This may allow an improved dynamic range transform to be performed inmany scenarios, and may specifically allow the dynamic range transformto compensate for specific characteristics of the tone mapping performedat the content creation side.

In some scenarios, the image processing device may thus take intoaccount both characteristics of the display for which the encoded imagehas been optimized and characteristics of the specific tone mapping.This may e.g. allow subjective and e.g. artistic tone mapping decisionsto be taken into account when transforming an image from one dynamicrange to another.

In accordance with an optional feature of the invention, the imagesignal further comprises a data field comprising dynamic range transformcontrol data; and the dynamic range processor is further arranged toperform the dynamic range transform in response to the dynamic rangetransform control data.

This may provide improved performance and/or functionality in manysystems. In particular, it may allow localized and targeted adaptationto specific dynamic range displays while still allowing the contentprovider side to retain some control over the resulting images.

The dynamic range transform control data may include data specifyingcharacteristics of the dynamic range transform which must and/or may beapplied and/or it may specify recommended characteristics of the dynamicrange transform.

In accordance with an optional feature of the invention, the dynamicrange transform control data comprises different dynamic range transformparameters for different display maximum luminance levels.

This may provide improved control and/or adaptation in many embodiments.In particular, it may allow the image processing device 103 to selectand apply appropriate control data for the specific dynamic range theoutput image is generated for.

In accordance with an optional feature of the invention, the dynamicrange transform control data comprises different tone mapping parametersfor different display maximum luminance levels, and the dynamic rangeprocessor is arranged to determine tone mapping parameters for thedynamic range transform in response to the different tone mappingparameters and a maximum luminance for the output image signal.

This may provide improved control and/or adaptation in many embodiments.In particular, it may allow the image processing device 103 to selectand apply appropriate control data for the specific dynamic range theoutput image is generated for. The tone mapping parameters mayspecifically provide parameters that must, may or are recommended forthe dynamic range transform.

In accordance with an optional feature of the invention, the dynamicrange transform control data comprises data defining a set of transformparameters that must be applied by the dynamic range transform.

This may allow a content provider side to retain control over imagesrendered on displays supported by the image processing device. This mayensure homogeneity between different rendering situations. The approachmay for example allow a content provider to ensure that the artisticimpression of the image will remain relatively unchanged when renderedon different displays.

In accordance with an optional feature of the invention, the dynamicrange transform control data comprises data defining limits fortransform parameters to be applied by the dynamic range transform.

This may provide improved operations and an improved user experience inmany embodiments. In particular, it may in many scenarios allow animproved trade-off between the desire of a content provider to retaincontrol over rendering of his/her content while allowing an end user tocustomize it to his/her preferences.

In accordance with an optional feature of the invention, the dynamicrange transform control data comprises different transform control datafor different image categories.

This may provide improved transformed images in many scenarios. Inparticular it may allow the dynamic range transform to be optimized forthe individual characteristics of the different images. For example,different dynamic range transforms may be applied to imagescorresponding to the main image, images corresponding to graphics,images corresponding to a background etc.

In accordance with an optional feature of the invention, a maximumluminance of the dynamic range of the first target display is no lessthan 1000 nits.

The image to be transformed may be an HDR image. The dynamic rangetransform may transform such an HDR image to another HDR image(associated with a display having a dynamic range of no less than 1000nits) having a different dynamic range. Thus, improved image quality maybe achieved by converting one HDR image for one dynamic range to anotherHDR image for another dynamic range (which may have a higher or lowerwhite point luminance).

In accordance with an optional feature of the invention, the imagesignal comprises a second encoded image and a second target displayreference, the second target display reference being indicative of adynamic range of a second target display for which the second encodedimage is encoded, the dynamic range of the second target display beingdifferent than the dynamic range of the first target display; and thedynamic range processor is arranged to apply the dynamic range transformto the second encoded image in response to the second target displayreference.

This may allow improved output quality in many scenarios. In particular,different transformations may be applied for the first encoded image andfor the second encoded image dependent on the differences of theassociated target displays (and typically dependent on how each of theserelate to the desired dynamic range of the output image).

In accordance with an optional feature of the invention, the imagedynamic range processor is arranged to generate the output image bycombining the first encoded image and the second encoded image.

This may provide improved image quality in many embodiments andscenarios. In some scenarios, the combination may be a selectioncombination where the combination is performed simply by selecting oneof the images.

In accordance with an optional feature of the invention, the imageprocessing apparatus further comprises: a receiver for receiving a datasignal from a display, the data signal comprising a data field whichcomprises a display dynamic range indication of the display, the displaydynamic range indication comprising at least one luminancespecification; and the dynamic range processor is arranged to apply thedynamic range transform to the first encoded image in response to thedisplay dynamic range indication.

This may allow improved image rendering in many embodiments.

In accordance with an optional feature of the invention, the dynamicrange processor is arranged to select between generating the outputimage as the first encoded image and generating the output image as atransformed image of the first encoded image in response to the firsttarget display reference.

This may allow improved image rendering in many embodiments and/or mayreduce the computational load. For example, if the end-user display hasa dynamic range which is very close to that for which the encoded imagehas been generated, improved quality of the rendered image willtypically be achieved if the received image is used directly. However,if the dynamic ranges are sufficiently different, improved quality isachieved by processing the image to adapt it to the different dynamicrange. In some embodiments, the dynamic range transform may simply beadapted switch between a null operation (using the first encoded imagedirectly) and applying a predetermined and fixed dynamic range transformif the target display reference is sufficiently different from the enduser display.

In accordance with an optional feature of the invention, the dynamicrange transform comprises a gamut transform.

This may allow an improved output image to be generated in manyembodiments and scenarios. In particular, it may allow a perceivedimproved color rendering and may for example compensate for changes incolor perception resulting from changes in the brightness of imageareas. In some embodiments the dynamic range transform may consist in agamut transform.

In accordance with an optional feature of the invention, the imageprocessing apparatus further comprises a control data transmitter fortransmitting dynamic range control data to a source of the image signal.

This may allow the source to adapt the image signal in response to thedynamic range control data. The dynamic range control data mayspecifically comprise an indication of a preferred dynamic range for theimage, and/or an indication of a dynamic range (e.g. white pointluminance and optionally EOTF or gamma function) for the end-userdisplay.

According to an aspect of the invention there is provided an imagesignal source apparatus comprising: a receiver for receiving a encodedimage; a generator for generating an image signal comprising the encodedimage and a target display reference indicative of a dynamic range of atarget display for which the encoded image is encoded; a transmitter fortransmitting the image signal

According to an aspect of the invention there is provided an imageprocessing method comprising:

receiving an image signal, the image signal comprising at least a firstencoded image and a first target display reference, the first targetdisplay reference being indicative of a dynamic range of a first targetdisplay for which the first encoded image is encoded;

generating an output image by applying a dynamic range transform to thefirst encoded image in response to the first target display reference;and

outputting an output image signal comprising the output image.

According to an aspect of the invention there is provided a method oftransmitting an image signal, the method comprising: receiving anencoded image; generating an image signal comprising the encoded imageand a target display reference indicative of a dynamic range of a targetdisplay for which the encoded image is encoded; and transmitting theimage signal

According to an aspect of the invention there is provided an imagesignal comprising at least a first encoded image and a first targetdisplay reference, the first target display reference being indicativeof a dynamic range of a first target display for which the first encodedimage is encoded.

These and other aspects, features and advantages of the invention willbe apparent from and elucidated with reference to the embodiment(s)described hereinafter.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the invention will be described, by way of example only,with reference to the drawings, in which

FIG. 1 is an illustration of examples of elements of an image renderingsystem in accordance with some embodiments of the invention;

FIG. 2 is an illustration of an example of elements of an imageprocessing apparatus;

FIG. 3 illustrates an example of a mapping for an image processingapparatus;

FIG. 4 illustrates an example of an Electro Optical Transfer Function(EOTF) for a display;

FIG. 5 illustrates an example of the model for presentation planes inthe HDMV-2D mode of the Blu-ray™ standard;

FIG. 6 illustrates an example of dynamic range processing for HDR andLDR images;

FIG. 7 illustrates an example of a mapping for an image processingapparatus;

FIG. 8-10 illustrate examples of images with different dynamic rangetransforms when presented on the same display;

FIG. 11 illustrates an example of a relationship between luminancevalues and possible mappings for an image processing apparatus;

FIG. 12 illustrates an example of a mapping for an image processingapparatus;

FIG. 13 illustrates an example of a mapping for an image processingapparatus;

FIG. 14 illustrates the structure of a graphics stream in accordancewith the Blu-ray™ standard;

FIG. 15 illustrates an example of the dynamic range processing for animage and an associated overlay graphics image;

FIG. 16 illustrates an example of the dynamic range processing for animage and graphics;

FIG. 17 is an illustration of an example of elements of an imageprocessing apparatus;

FIG. 18 illustrates an example of a mapping for an image processingapparatus;

FIG. 19 is an illustration of an example of elements of an imageprocessing apparatus;

FIG. 20 illustrates an example of a mapping for an image processingapparatus;

FIG. 21 is an illustration of an example of elements of a display inaccordance with some embodiments of the invention;

FIG. 22 is an illustration of an example of elements of an imageprocessing apparatus; and

FIG. 23 schematically illustrates a generation of an 8 bit imageencoding a HDR image by means of an encoding apparatus

DETAILED DESCRIPTION OF SOME EMBODIMENTS OF THE INVENTION

FIG. 1 illustrates an example of an audio visual distribution path. Inthe example, a content provider apparatus 101 generates an audio visualcontent signal for an audiovisual content item, such as e.g. a film, atelevision program etc. The content provider apparatus 101 mayspecifically encode the audiovisual content in accordance with asuitable encoding format and color representation. In particular, thecontent provider apparatus 101 may encode the images of a video sequenceof the audiovisual content item in accordance with a suitablerepresentation such as e.g. YCrCb. The content provider apparatus 101may be considered to represent a production and distribution house whichcreates and broadcasts the content.

The audio visual content signal is then distributed to an imageprocessing device 103 via a distribution path 105. The image processingdevice 103 may for example be a set-top box residing with a specificconsumer of the content item, such as e.g. a Personal Video Recorder, aBlu-ray™ player, a network (e.g. Internet) streaming device, a satelliteor terrestrial television receiver, etc.

The audio-visual content is encoded and distributed from the contentprovider apparatus 101 through a medium, which may e.g. consist ofpackaged medium or a communication medium. It then reaches a sourcedevice in the form of the image processing device 103 which comprisesfunctionality for decoding and playing back the content.

It will be appreciated that the distribution path 105 may be anydistribution path and via any medium or using any suitable communicationstandard. Further, the distribution path need not be real time but mayinclude permanent or temporary storage. For example, the distributionpath may include the Internet, satellite, cable or terrestrialbroadcasting, a mobile or fixed communication network etc., or storageon physically distributed media such as DVD or Blu-ray Disc™ or a memorycard etc.

The image processing device 103 is coupled to a display 107 via acommunication path 109. The image processing device 103 generates adisplay signal representing the audiovisual content item. Thus, thesource device streams the decoded content to a sink device, which may bea TV or another device which converts the digital signals to a physicalrepresentation.

The image processing device 103 may perform e.g. image enhancement orsignal processing algorithms on the data and may specifically decode andre-encode the (processed) audiovisual signal. The re-encoding mayspecifically be to a different encoding or representation format thanfor the received signal.

The system of FIG. 1 is in some embodiments arranged to provide HighDynamic Range (HDR) video information to the display 107 and in otherembodiments or scenarios is arranged to provide a Low Dynamic Range(LDR) image to the display 107. Further, in order to provide e.g.improved backwards compatibility, it may in some scenarios be able toprovide both an LDR and an HDR image depending on the display on whichit is displayed. Specifically, the system is able to communicate/distribute image signals relating to both LDR and HDR images.

Conventional displays typically use an LDR representation. Typicallysuch LDR representations are provided by a three component 8 bitrepresentation related to specified primaries. For example, an RGB colorrepresentation may be provided by three 8 bit samples referenced to aRed, Green, and Blue primary respectfully. Another representation usesone luma component and two chroma components (such as YCrCb). These LDRrepresentations correspond to a given brightness or luminance range.

HDR specifically allows for significantly brighter images (or imageareas) to be presented appropriately on HDR displays. Indeed, an HDRimage displayed on an HDR display may provide a substantially brighterwhite than can be provided by the corresponding LDR image presented onan LDR display. Indeed, an HDR display may allow typically at least afour times brighter white than an LDR display. The brightness mayspecifically be measured relative to the darkest black that can berepresented or may be measured relative to a given grey or black level.

The LDR image may specifically correspond to specific displayparameters, such as a fixed bit resolution related to a specific set ofprimaries and/or a specific white point. For example, 8-bits may beprovided for a given set of RGB primaries and e.g. a white point of 500Cd/m2. The HDR image is an image which includes data that should berendered above these restrictions. In particular, a brightness may bemore than four times brighter than the white point (e.g. 2000 Cd/m2) ormore.

High dynamic range pixel values have a luminance contrast range(brightest luminance in the set of pixels divided by darkest luminance)which is (much) larger than a range that can be faithfully displayed onthe displays standardized in the NTSC and MPEG-2 era (with its typicalRGB primaries, and a D65 white with maximum driving level [255, 255,255] a reference brightness of e.g. 500 nit or below). Typically forsuch a reference display 8 bits suffice to display all grey valuesbetween approximately 500 nit and approximately 0.5 nit (i.e. withcontrast range 1000:1 or below) in visually small steps, whereas HDRimages are encoded with a higher bit word, e.g. 10 bit (which is alsocaptured by a camera with a larger well depth and DAC, e.g. 14 bit). Inparticular, HDR images typically contain many pixel values (of brightimage objects) above a scene white. In particular, several pixels arebrighter than two times a scene white. This scene white may typically beequated with the white of the NTSC/MPEG-2 reference display.

The number of bits used for the HDR images X may typically be largerthan or equal to the number of bits Y used for LDR images (X maytypically be e.g. 10 or 12, or 14 bit (per color channel if several ofthe channels are used), and Y may e.g. be 8, or 10 bits). Atransformation/mapping may be required to fit pixels in a smaller range,e.g. a compressive scaling. Typically, a non-linear transformation maybe involved, e.g. a logarithmic encoding may encode (as lumas) a farlarger luminance range in an X-bit word than a linear encoding, be itthat the luminance difference steps from one value to the next are thennot equidistant, but nor are they required to be so for the human visualsystem.

It should be noted that the difference between LDR and HDR images is notmerely that a larger number of bits are used for HDR images than for LDRimages. Rather, HDR images cover a larger luminance range than LDRimages and typically have a higher maximum luminance value, i.e. ahigher white point. Indeed, whereas LDR images have a maximum luminance(white) point corresponding to no more than 500 nits, HDR images have amaximum luminance (white) point corresponding to more than 500 nits, andoften no less than 1000 nits, 2000 nits or even 4000 nits or higher.Thus, an HDR image does not merely use more bits corresponding to ahigher granularity or improved quantization but rather corresponds to alarger actual luminance range. Thus, the brightest possible pixel valuegenerally corresponds to a luminance/light output which is higher for anHDR image than for an LDR image. Indeed, HDR and LDR images may use thesame number of bits but with the HDR image values being referenced to alarger luminance dynamic range/ brighter maximum luminance than the LDRimage values (and thus with the HDR images being represented with a morecoarse quantization on a luminance scale).

Ideally, the content provided by the content provider apparatus 101 willbe captured and encoded with reference to a luminance range that matchesthe luminance range of the display 107. However, in practical systemsthe content may be rendered on many different displays with manydifferent characteristics, and/or may be encoded according to standardsthat are based on luminance ranges which differ from the luminance rangeof the specific display 107. Furthermore, the content may not originallybe captured by a capturing device or approach that exactly matches theluminance range of the display.

Accordingly, the support of HDR in a content system typically requiressome transformation or conversion between different luminance ranges.For example, if an LDR image is received and should be presented on anHDR display, a conversion from LDR to HDR should be performed. If an HDRimage is received and should be presented on an LDR display, aconversion from HDR to LDR should be performed. Such conversions aretypically rather complex and do not merely equate to a simple scaling ofthe luminance ranges as such a scaling would result in an image whichwould be perceived as unnaturally looking. Rather complextransformations are typically used and these transformations are oftenreferred to using the term tone mapping.

In principle, such luminance transformations could be performed at threedifferent places in the content distribution system.

One option is to perform it at the content provider apparatus 101.Typically, this may allow the same luminance transformation operation tobe distributed to multiple displays thereby allowing a singletransformation to be used for many users. This may allow and justifycomplex, manual and resource demanding tone mapping to be performed,e.g. by skilled tone mapping experts. Indeed, this may provide asubjectively optimized image for a given luminance range, often referredto as an artistic tone mapping. However, such an approach is veryresource demanding and is not feasible for application to many displays.Furthermore, a separate image stream is required for each supportedluminance range resulting in a very high communication resource beingneeded which is impractical for many systems.

Another option is to perform the luminance transform in the imageprocessing device 103. However, as the general user is not skilled inluminance transforms and since the required effort renders itimpractical to perform manual adaptation (especially for moving images,such as video clips, films etc), the transformation should preferably beautomatic. However, such transforms are conventionally not able toprovide optimum images. In particular, the optimum transform may dependon the specific type of content, the intended characteristics of theimage (e.g. different transforms may be appropriate for a scene intendedto be dark and menacing and a scene which is just intended to be dark toindicate a night time scene). Furthermore, the content originator may beconcerned about the potential impact of such automatic transforms andmay be reluctant to lose the control over how the content may bepresented in different scenarios. Also, the optimum transformation willtypically depend on the exact characteristics of the display 107 and atransformation based on an assumed, nominal or standard display willtypically result in suboptimal transforms.

The transform may possibly also be performed in the display 107.

In the system of FIG. 1 the image processing device 103 comprisesfunctionality for performing a luminance dynamic range transform on animage (or set of images, such as e.g. a video sequence) received fromthe content processing device 103 in order to increase the dynamic rangethereof. In particular, the image processing device 103 receives animage from the content provider apparatus 101 then processes the imageto generate a higher dynamic range image. Specifically, the receivedimage may be an LDR image which is converted into an HDR image byapplying the luminance dynamic range transform to increase the dynamicrange. The transformed image can then be output to the display 107 beingan HDR display thereby resulting in the originally received LDR imagebeing converted into a rendered HDR image. A dynamic range transform maymap luminance values of (at least part) of an input image associatedwith one dynamic range to luminance values for (at least part) of anoutput image associated with a different dynamic range.

In another scenario, the image processing device 103 may receive animage from the content provider apparatus 101 and then process the imageto generate a lower dynamic range image. Specifically, the receivedimage may be an HDR image which is converted into an LDR image byapplying the luminance dynamic range transform to decrease the dynamicrange. The transformed image can then be output to the display 107 beingan LDR display thereby resulting in the originally received HDR imagebeing converted into a rendered LDR image.

In the system of FIG. 1, the dynamic range transform is adapted independence on information received from the content provider apparatus101 and/or the display 107. Thus, in the system, the dynamic rangetransform is not merely a locally performed operation in the imageprocessing device 103 but may also be dependent on characteristics,properties or information from the content provider apparatus 101 and/orthe display 107.

First the system of FIG. 1 will be described with reference to asituation where the dynamic range transform is based on informationprovided to the image processing device 103 from the content providerapparatus 101.

FIG. 2 illustrates an example of elements of the image processing device103 of FIG. 1.

The image processing device 103 comprises a receiver 201 which receivesan image signal from the content provider apparatus 101. The imagesignal comprises one or more encoded images. In many scenarios the imagesignal may be a video signal comprising an encoded video sequence, i.e.a sequence of images. It will be appreciated that any suitable encodingof the image(s) may be used including for example JPEG image coding,MPEG video coding, etc. The encoded image is represented by pixel valueswhich for each pixel of the image represents the corresponding lightoutput for the pixel (or for individual color channel subpixel). Thepixel values may be provided in accordance with any suitable colorrepresentation such as e.g. RGB, YUV etc.

The image signal furthermore comprises a target display reference whichis indicative of a dynamic range of a target display for which the firstencoded image is encoded. Thus, the target display reference provides areference for the encoded image which reflects the dynamic range forwhich the received image has been constructed. The target displayreference may indicate the luminances for which the tone mapping at thecontent provider apparatus 101 has been designed, and specificallyoptimized for.

The content provider apparatus 101 is thus arranged to generate an imagesignal which not only includes the encoded image itself but also atarget display reference which represents the dynamic range of thedisplay for which the encoded signal has been generated. The contentprovider apparatus 101 may specifically receive the encoded image froman internal or external source. E.g. the image may be provided as aresult of a manual tone grading which optimizes the encoded image for aspecific display. In addition, the content provider apparatus 101 mayobtain information of the specific display that has been used for theoptimization, e.g. via display information that has been automaticallycommunicated to the content provider apparatus 101 from the display(e.g. the content provider apparatus 101 may also include thefunctionality required for supporting the manual tone mapping and may beconnected to the target/reference display used for this tone mapping).As another example, the encoded tone mapped image may be received on amedium on which the properties of the associated display are alsostored. As yet another example, the content provider apparatus 101 mayreceive information of a characteristic of the target display by amanual user input.

The content provider apparatus 101 may in response to such informationgenerate an image signal which comprises both the encoded image(s) andtarget display reference which indicates a dynamic range of the targetdisplay that was used for the tone mapping. E.g. a data valuecorresponding to an identification of a white point luminance andoptionally an Electro Optical Transfer Function corresponding to that ofthe target display may be included in the image signal by the contentprovider apparatus 101.

The image processing device 103 furthermore comprises a dynamic rangeprocessor 203 which applies the dynamic range transform to the receivedencoded image in order to generate an output image with a higher dynamicrange, i.e. which corresponds to a larger range of output luminanceswhen the image is rendered. Specifically, the input encoded image may bean image which is encoded for a LDR display with a maximum luminancewhite point of 500 nits and this may be transformed into an HDR outputimage with a maximum luminance white point of e.g. 1000 or 2000 nits.Typically, the dynamic range transform may also increase the number ofbits used to represent each value but it will be appreciated that thisis not essential and that in some embodiments the same number of bits(or indeed even fewer bits) may be used for the output image than forthe input image. As another example, the input encoded image may be animage which is encoded for a HDR display with a maximum white pointluminance of 2000 nits and this may be transformed into an LDR outputimage with a maximum white point luminance of e.g. 500 nits. Such adynamic range reduction transform may also include a reduction in thenumber of bits used for the pixel values. The dynamic range transform isperformed in response to the target display reference and thus may beadapted to take into account not only the desired output luminance rangebut also the luminance range for which the received image has beenencoded. For example, the system may adapt the dynamic range transformsuch that a transform to generate an output image for 1000 nits will bedifferent depending on whether the input image is generated for a 300nits or 500 nits image. This may result in a substantially improvedoutput image.

Indeed, in some embodiments the input image may itself be an HDR image,such as e.g. a 1000 nits image. The optimal transformation of such animage into respectively a 2000 nits image and a 5000 nits image willtypically be different and the provision of a target display referencemay allow the image processing device 103 to optimize the dynamic rangetransform for the specific situation, thereby providing a substantiallyimproved image for the specific display characteristics. Indeed, if thedisplay is a 500 nits display, the dynamic range transform shouldperform a dynamic range compression rather than expansion. Theapproaches may be particularly advantageous in inhomogeneous contentdistribution systems such as e.g. what is increasingly perceived forfuture television systems. Indeed the (peak) brightness of HDR LCD/LEDTVs is currently rapidly increasing and in the near future, displayswith a wide variety of (peak) brightness are expected to coexist in themarket. Brighter pictures look nicer on the TV screen and a brighter TVsells better in the shop. On the other hand, “low end” displays innotebooks, tablets and smart phones are also becoming very popular andare also used for the rendering of e.g. TV content.

Since the display brightness (and typically the electro-optical transferfunction that specifies how a display converts input pixel (color)driving values into light values which then provides a particularpsychovisual impression to the viewer) is no longer known at the contentgeneration side (and which is furthermore generally different from thereference monitor for which the content was intended/graded), it becomeschallenging to provide the best/optimal picture quality on the display.Furthermore, whereas some variations in display brightness may haveexisted in the past, this variation was relatively minor and theassumption of a known fixed brightness did not introduce significantdegradations (and could often be compensated manually be a user, e.g. bysetting the brightness and/or contrast of a display).

However, due to the substantial increase in the variety of displays(smart phones, tablets, laptops, PC monitors, CRT displays, traditionalLCD TV displays and bright HDR displays), the characteristics(especially brightness and contrast) of the displays used for renderingexhibit an enormous variation. For example, the contrast and peakluminance of state-of-the-art high-end display systems is continuallyincreasing and new prototype displays have been developed with a peakluminance as high as 5000 cd/m2 and contrast ratios of 5-6 orders ofmagnitude. On the other hand displays being used in, for example smartphones and tablets, are becoming more and more popular but haverelatively low performance characteristics.

As mentioned previously content, such as video for movies etc, isprocessed at the content creation side to provide desired renderedimages. For example, when a movie is issued for general distribution(such as by DVD or Blu-ray™) the producers/studio typically adapts andcustomizes the images for optimal appearance on a specific display. Sucha process is often referred to as color grading and tone mapping. Tonemapping may be considered as a non-linear mapping of a luma value of aninput pixel to the luma value of an output pixel. The tone mapping isperformed in order to match the video to the characteristics of thedisplay, viewing conditions and subjective preferences. In case of localtone mapping, the processing varies depending on the position of thepixel within an image. In case of global tone mapping, the sameprocessing is applied to all pixels.

For example, when converting content to be suitable for general consumerdistribution, tone mapping is often performed to provide a desiredoutput on a standard LDR display. This may be performed manually bycolor grading experts that balance many picture quality aspects tocreate the desired ‘look’ for the storyline. This may involve balancingregional and local contrasts, sometimes even deliberately clippingpixels. Thus, typically the tone mapping at this stage is not merely asimple automated conversion but is typically a manual, subjective andoften artistic conversion.

If the content were graded for an HDR target display rather than for anLDR target display, the outcome of the tone mapping would typically bevery different. Thus, when merely rendering the video content encodedfor an LDR display on a HDR display, the resulting images will differsubstantially from the optimal image. Similarly, if an HDR optimizedimage is merely rendered on an LDR display, a significant perceivedimage quality reduction may occur.

This issue is in the system of FIG. 1 addressed by the dynamic rangetransform being performed in the image processing device 103 but beingbased on information received preferably both from the content providerapparatus 101 and the display 107. In this way, the dynamic rangetransform (specifically a tone mapping algorithm) can be adapted toconsider the characteristics of the tone mapping that was performed inthe content provider apparatus 101 and to the specific luminance rangeof the display 107. Specifically, the tone mapping performed at theimage processing device 103 can be dependent on the target display forwhich the tone mapping is performed at the content generation side.

The content provider apparatus 101 provides a target display referenceto the image processing device 103 (either separately to or integratedwith the encoded image, i.e. the image signal may be made up of twoseparate data communications). The target display reference mayspecifically include or be a white point luminance of the targetdisplay.

For example, for a relatively low complexity system, the contentprovider apparatus 101 may simply transmit an indication of the whitepoint luminance of the target display for each the encoded image (video)that has been encoded. E.g., data may be communicated that indicates thenumber of nits available at the target display. The dynamic rangetransform can then adapt the transformation based on the number of nits.For example, if the image processing device 103 is performing a dynamicrange transform to generate an output image for a 2000 nits display, theknowledge of whether the input image is tone mapped to a display of 500nits or one of 1000 bits can be used to optimize the dynamic rangetransform performed at the image processing device 103. In bothscenarios, the dynamic range transform may apply a non-linear transformbut this non-linear transform may have different characteristics for thetwo scenarios, i.e. dependent on the white point of the target displayused for the tone mapping at the content provision side.

For example, the following mapping between received LDR image pixelstone mapped for a 500 nits target display and the output HDR imagepixels for a 2000 nits end-user display may be performed:

-   -   0-200 nits→0-200 nits    -   200-300 nits→200-600 nits    -   300-400 nits→600-1000 nits    -   400-500 nits→1000-2000 nits

However, for a target display of 1000 nits, the following mapping mayinstead be performed:

-   -   0-200 nits→0-200 nits    -   200-700 nits→200-1000 nits    -   700-1000 nits→1000-2000 nits

Thus, in terms of relative values (percentage of full mapping), the twodifferent mappings may be as shown in FIG. 3 where the relationshipbetween the percentage of white level for the input image on the x-axisrelative to the percentage of white level for the output image on they-axis is shown for respectively a 500 nit target display 301 and a 1000nit target display. In the example, two very different non-linear tonemappings are applied for the same user display depending on the targetreference display that was used/assumed at the content provision side.

It will be appreciated that the same mappings can be used for mappingfrom a 2000 nits optimized image to a 500 or 1000 nits optimized imageby interchanging the axes (corresponding to applying an inverse mappingof that described above). It will also be appreciated that the mappingto e.g. a 500 nits optimized image may be adapted depending on whetherthe input image is a 1000, 2000 or 4000 nits optimized image.

In some embodiments, the target display reference may alternatively oradditionally comprise an Electro Optical Transfer Function indicationfor the target display. For example, a gamma indication for the targetdisplay may be included.

The Electro-Optical Transfer Function (EOTF) of a display describes therelationship between input (driving) luma value (Y′) and outputluminance (Y) for the display. This conversion function depends on manycharacteristics of the display. Also user settings like brightness andcontrast may have great influence on this function. FIG. 4 illustrates atypical example of an EOTF for an 8 bit (256 level) input value.

The communication of an EOTF of the target display can provide anadvantageous characterization of the target or reference display used togenerate the encoded image or video. This characterization can then beused at the image processing device 103 to adapt the dynamic rangetransform to the differences between the characteristics of the targetdisplay and the end-user display. For example, the dynamic rangetransform may include a compensation that inverts a ratio between theEOTFs of the target/reference display and the end-user display.

It will be appreciated that there are many ways to characterize an EOTF.One possibility is to provide a set of sample values of the EOTF. Theimage processing device 103 may then interpolate between the samplepoints, e.g. using simple linear interpolation. Another possibility isto provide a specific model of grey scale/contrast behavior of thedisplay at least over a part of the display range. As another example,the content provider apparatus 101 may communicate a specificmathematical function characterizing the EOTF. In some scenarios, a setof target displays may be predefined with the associated parameters ofthe model/function being stored locally in the image processing device103. In that case the content provider apparatus 101 may onlycommunicate the identification code of the target display to the imageprocessing device 103.

As yet another example, an underlying mathematical function may bepredetermined and the target display indication may comprise parametersfor adapting the predetermined function to describe the specific targetdisplay EOTF. For example, the EOTF may be characterized by a gammafunction as used for conventional displays, and the target displayindication may provide a specific gamma for the target display.

In many systems, the target display indication may comprise or consistin a maximum luminance and a gamma of the target display. Thus,specifically, the characterization of the EOTF may be provided by twovalues, namely the gamma and the white point/ maximum luminance Thefollowing descriptions will focus on such a scenario.

The description will also focus on embodiments wherein the distributionsystem is according to the Blu-ray™ standard. Blu-ray™ is a family ofAudio/Video/Data distribution formats based on optical disc technology.BD-ROM™ is the acronym for Blu-ray Disc Read-only format. This format ispredominantly used for distribution of high definition video (2D and 3D)and high quality audio.

A BD-ROM™ player features two modes of operation: HDMV and BD-J. At anypoint in time the player is either in HDMV mode or BD-J mode. Profile 5Blu-ray™ players feature rendering of 3D stereoscopic Video/Graphicsnext to the standard 2D Video/Graphics rendering. As an example FIG. 5shows the model for presentation planes in the HDMV-2D mode. As aspecific example of the system of FIG. 1, the image signal may be avideo signal encoded on a BDROM™ and thus the image processing device103 may specifically be a Blu-ray™ player. The encoded video may be theprimary or optionally the secondary video content on the disc. Theprimary video is typically the actual movie in 2D or possibly in 3Dstereoscopic format.

In order to achieve optimal picture quality in the BDROM™ system, thesystem of FIG. 1 uses an augmentation to the BDROM™ specification whichallows for transmission of a target display parameters. This datatogether with the assumed or actual information of the end-user display,is then used by the BDROM™ player to perform the dynamic rangetransform. Specifically, the BDROM™ player (the image processing device103) may perform additional video tone mapping or other processingdepending on the characteristics of the target display and/or theend-user display.

One option for transmitting information on the parameters of the targetdisplay is by embedding data indicative of these parameters values inthe BDROM™ data on the disc. An extension data structure in the playlistfile (xxxxx.mpls) can be used for this. This extension data structurewill have a unique and new identification. Incompatible legacy BDROM™players will be ignorant of this new data structure and will merelyignore it. This will guarantee backward compatibility. A possibleimplementation of the syntax and semantics of such aTarget_Display_descriptor is shown below.

No. of Syntax bits Mnemonics Target_Display_Descriptor ( ) {Abs_Max_Luminance 8 uimsbf Gamma (or grey value behavior 8 uimsbf modelof the display e.g. EOTF) }

In this example, Abs_Max_Luminance is a parameter with a value e.g.between 0 and 255 that indicates the Absolute maximum luminance/whitepoint of the target display according to:

Absolute maximum luminance in cd/m2=Abs_Max_Luminance [bit0-4]×10^(Abs)^(_) ^(Max) ^(_) ^(Luminance [bits-)7].

It will be appreciated that other amounts of bits for mantissa orexponent may of course be used.

Gamma is a parameter with a value e.g. between 0 and 255 that indicatesthe gamma of the target display according to:

Gamma of the Target Display EOTF=Gamma/25.

Thus, in this example a target display reference is provided to theimage processing device 103 by the BDROM™ including an absolute maximumluminance and a gamma value for the target display for which the videosignal has been generated. The image processing device 103 then usesthis information when performing an automatic dynamic range transform toincrease or decrease the dynamic range of the video signal for ahigher/lower luminance end-user display.

It will be appreciated that many different dynamic range transforms arepossible and that many different ways of adapting such dynamic rangetransforms based on the target display references may be used. In thefollowing, various examples are provided but it will be appreciated thatother approaches may be used in other embodiments.

Firstly, the difference in the optimum mapping of a given original imageto respectively an LDR and an HDR image may be illustrated by FIG. 6which shows an example of the different tone mapping that may be usedfor an LDR display (lower part of the figure) and an HDR display (upperpart of the figure). The original image is the same for both LDR andHDR. The histogram of this image is shown at the left of FIG. 6. Itshows that most pixels have luma values in the low-mid range. Thehistogram also shows a second, small peak at high luma values (e.g.headlights of a car or a flashlight).

In this example, tone mapping is represented by three successiveprocessing steps:

-   Clipping: Mapping of luma values in the low and high range to a    limited number of output luma values.-   Expanding: Adapting the dynamic range to the desired luma dynamic    range.-   Brightness: Adapting the average luminance level for optimal    brightness.

In the LDR case, the luma range is mapped to a luminance range of an LDRdisplay. The dynamic range of the original image is much larger and thusthe original image is severely clipped in order to accommodate for thelimited dynamic range of the display.

In the HDR case (upper part of the figure) the clipping can be lesssevere because the dynamic range of the display is an order of magnitudelarger than for the LDR display.

FIG. 6 shows the histogram after each processing steps as well as thehistogram of the image shown on the LDR and HDR display respectively. Inparticular, the rightmost histograms illustrate the LDR tone mappedimage when shown on a HDR display and vice versa. In the first case theimage will be too bright and the low and high range luma values willlose too much detail. In the second case the image will be too dark andthe middle range luma values lose too much detail and contrast.

As can be seen, merely presenting a (luminance scaled version of) theLDR optimized image on an HDR display (or vice versa) may substantiallyreduce image quality, and therefore the image processing device 103 mayperform a dynamic range transform to increase the image quality.Furthermore, since the optimization performed at the studio dependsstrongly on the characteristics of the display for which theoptimization has been performed, the optimum dynamic range transform tobe performed by the image processing device 103 does not merely dependon the end-user display but also depends on the reference display.Accordingly, the target display reference provided to the imageprocessing device 103 allows the image processing device 103 to performthe desired dynamic range transform not merely based on the assumed orknown characteristics of the end-user display, but also based on theactual display used at the content provider side. Indeed, it can beconsidered that the provision of the target display reference allows theimage processing device 103 to partially or fully reverse some of thetone mapping performed at the studio side thereby allowing estimation ofcharacteristics of the original image. Based on this estimation, theimage processing device 103 can then apply a desired tone mappingoptimized for the specific dynamic range characteristics of the end-userHDR display.

It will be appreciated that the image processing device 103 typicallydoes not seek to perform a specific inverse tone mapping to recreate theoriginal signal followed by a tone mapping suitable for the specificend-user display. Indeed, typically the dynamic range transform will notprovide sufficient information to perform such inverse tone mapping andthe tone mapping performed by the content provider may often bepartially irreversible.

However, the image processing device 103 may perform a dynamic rangetransform which seeks to adapt the received image by the dynamic rangetransform providing a result that may be a (possibly very coarse)approximation of the more theoretical operation of an inverse tonemapping to generate the original image followed by an optimized tonemapping of the original image to the specific desired dynamic range.Thus, the image processing device 103 may simply apply e.g. a simplemapping from luma values of the input to the dynamic range transform toappropriate luma values at the output of the transformation. However,this mapping not only reflects the desired tone mapping of the originalimage for the given end-user display but also depends on the actual tonemapping already performed at the content provider apparatus 101.Therefore, the image processing device 103 may use the dynamic rangetransform to adapt the applied transform to take into account and adaptto the tone mapping that has already been performed.

As an example, the image processing device 103 may be arranged toprovide an output image for display on an HDR image with a predeterminedmaximum luminance (say 4000 nits). The received image/video may be tonemapped for an LDR display of 500 nits. This tone mapping has thusoptimized the image for a given maximum luminance and gamma. As aspecific example, the gamma function may be as curve 701 of FIG. 7 andthe resulting image when presented on a 500 nits display may be as FIG.8.

When this image is to be presented on an HDR display of e.g. 4000 nits,it is often desirable that the light output for dark areas does notchange substantially whereas the light output for bright areas should beincreased very substantially. Thus, a very different relationshipbetween (linear) luminance values and actual drive values are required.Specifically, a substantially improved image would have been generatedfor an HDR image if the mapping curve 703 of FIG. 7 had been used, i.e.if a higher gamma had been applied at the content side tone mapping.However, this higher mapping will on a 500 nits display result in imagesthat appear to be too dark as illustrated in FIG. 9.

In the system, the image processing device 103 is informed of the gammavalue for the target display at the content side, and it can thus derivecurve 701. Furthermore, the desired curve 703 is known as it depends onthe display dynamic range for which the output image is generated (whiche.g. may be provided to the image processing device 103 from the display107 or may be assumed/ predetermined). Thus, the image processing device103 can apply a transformation to each pixel luminance valuecorresponding to the conversion from curve 701 to curve 703. In thisway, the image processing device 103 can thus proceed to use the targetdisplay reference provided from the content provider apparatus 101 toapply a dynamic range transform which converts the generated outputsignal from one suitable for an LDR display to one suitable for an HDRdisplay.

It will be appreciated that the same considerations may apply whenperforming the dynamic range transform to reduce the dynamic range. Forexample, if the received content is to be displayed on a low quality,low luminance display, such as a mobile phone display, the preferredgamma for the mapping curve may be as indicated by curve 705 of FIG. 7,i.e. a gamma of less than one may be preferred. When presented on anormal 500 nits LDR, a corresponding image would appear to be too brightand have too little contrast as indicated by FIG. 10, and indeed thescenario would be even worse for an HDR display.

Thus, if the image processing device 103 is generating an image for sucha low brightness display, it may proceed to perform a dynamic rangetransform that reduces the dynamic range by adjusting the luminancevalues for the differences in the gamma between curve 701 and 705.

As another example, if the content provider apparatus 101 provides animage intended for a low brightness/dynamic range display andaccordingly an image which is encoded according to curve 705, the imageprocessing device 103 can use the knowledge of this gamma provided bythe dynamic range transform to transform the received values to valuessuitable for either a 500 nits display by adapting for the differencebetween curves 705 and 701, or for a 4000 nits display by adapting forthe difference between curves 705 and 703.

Thus, the provision of a dynamic range transform indicating a maximumluminance/ white point luminance and a gamma value assumed for thetarget display allows the image processing device 103 to convert thereceived image to a gamma value suitable for the specific brightnessluminance value of the display on which the image is to be rendered.

In some systems, the target display reference may comprise a tonemapping indication representing a tone mapping used to generate thefirst encoded video stream for the first target display.

In some systems, the target display reference may directly provideinformation of some of the specific tone mapping that has been performedat the content provider side.

For example, the target display reference may include information thatdefines the white point luminance and gamma for which the LDR (or HDR)image has been generated, i.e. the display for which the tone mappinghas been performed. However, in addition, the target display referencemay provide some specific information that e.g. defines some of theinformation lost in the tone mapping that has been performed at thecontent provider side.

E.g. in the example of FIG. 6, an LDR tone mapped image corresponding tothe clipped image may be received by the image processing device 103.The image processing device 103 may apply a dynamic range transformwhich maps this to the appropriate dynamic range and non-linear relationbased on information of the target display gamma and white point.However, in order to provide an improved adaptation, the severe clippingused for the LDR image should preferably be translated into a lesssevere clipping (or indeed in some scenarios to no clipping).Accordingly, the content provider apparatus 101 may provide additionalinformation that identifies the specific clipping that has beenperformed for the LDR image by the content provider thereby allowing theclipping to be partially or fully reversed. For example, the dynamicrange transform may define the range which has been clipped and theimage processing device 103 may accordingly distribute the clippedvalues over this range in accordance with a suitable algorithm (e.g.identifying an area of clipped values (such as an explosion) andgenerating an increasing brightness towards the center of this area).

The dynamic range transform may alternatively or additionally provideinformation that defines an additional tone mapping that has beenperformed at the content provider side. For example, a relativelystandard tone mapping may be performed for most images of a movie orother video sequence. The image processing device 103 may, based on thegamma and white point luminance, convert such a tone mapped image to adesired (higher or lower) dynamic range image using a dynamic rangetransform that assumes a standard tone mapping at the content providerside. However, for some images the content provider may have performed adedicated and subjective tone mapping. For example, the color grader maydesire a specific artistic effect or quality for some images, such ase.g. a fine graduation or color cast for dark images of a tensesituation (say in a horror move) or a specific effect for dream likescenes. This tone mapping can be characterized by data in the targetdisplay reference thereby allowing the image processing device 103 toadapt the dynamic range transform to the specific tone mapping that hasbeen applied.

Thus, specifically, in some scenarios additional/ modified tone mappingis performed at the content provider side to generate a specific looksuch that the image is modified relative to that which would be expectedby a fixed adaption to the naked electro-optical behavior of the targetdisplay. The data provided by the content provider apparatus 101 mayspecify a desired look compared to the reference display and this can bythe image processing device 103 be used to actually generate the desiredoptical behavior given all factors (e.g. whereas a blind coding in theinput signal could accidentally end up below the reflected surroundlight such that it can no longer be compensated according to the encodedcontent provider side behavior).

As an example, if it is known that the gamma of the target display islow for the darker values, it is for such a (reference) display possibleto fine tune the look of, say , horror scenes. E.g. the image may becompensated by an extra luminance boosting so that the image stillappear darkish but at least with some object structure still beingvisible.

As an example, together with the gamma and white point luminance of thereference target, the color grader at the content provision side mayprovide some (additional) information about the artistic impression ofcertain regions and/or images. For example, for a given EOTF, thecontent provider may indicate that a certain area is desired to haveincreased brightness for better visibility, or decreased contrast toprovide a foggy look etc. Thus, together with an EOTF (e.g. representedby gamma and white point luminance) the target display reference mayindicate boundaries of a local/partial display luminance range andprovide dynamic range transform data that provides more precise info onthe preferred allocation of the grey levels therefor.

In some embodiments, the dynamic range processor (203) may be arrangedto select between generating the output image as the received encodedimage and generating the output image as a transformed image of thefirst encoded image in response to the target display reference.

Specifically, if the white point luminance indicated by the targetdisplay reference is sufficiently close to the white point luminance ofthe end-user display, the dynamic range transform may simply consist innot performing any processing on the received encoded image, i.e. theinput image may simply be used as the output image. However, if thewhite point luminance indicated by the target display reference isdifferent to the white point luminance of the end-user display, thedynamic range transform may modify the received image in accordance witha suitable mapping of the received image pixels to output image pixels.In such cases, the mapping may be adapted depending on the targetdisplay reference. In other examples, one or more predetermined mappingsmay be used.

For example, the image processing device 103 may include a predeterminedfirst mapping which has been determined to provide a suitable outputimage for a doubling in the white point luminance level and apredetermined second mapping which has been determined to provide asuitable output image for a halving of the white point luminance level.In such an example, the image processing device 103 may select betweenthe first mapping, the second mapping, and a unity mapping dependent onthe white point luminance of the target display reference and the whitepoint of the end-user display. The image processing device 103 mayspecifically select the mapping which most closely corresponds to theratio between the target display reference white point luminance and theend-user display white point luminance.

For example, if an input image is received with a target displayreference indicating that it has been optimized for a 500 nits displayand the end-user display is a 1000 nits display, the image processingdevice 103 will select the first mapping. If instead, the target displayreference indicates that the input image has been optimized for a 1000nits display, the image processing device 103 will select the unitymapping (i.e. use the input image directly). If the target displayreference indicates that it has been optimized for a 2000 nits display,the image processing device 103 will select the second mapping.

If in-between values for the white point luminance of the target displayare received, the image processing device 103 may select the mappingclosest to the ratio between the white point luminances, or may e.g.interpolate between the mappings.

In some embodiments, the dynamic range transform may comprise or consistin a gamut transform. Thus, in some embodiments, the dynamic rangeprocessor 203 may modify chromaticities of the rendered image dependingon the target display reference. For example, when a received HDR imageis rendered on an LDR display the compression may result in a blanderimage with fewer variations and gradations in individual image objects.The dynamic range transform may compensate for such reductions byincreasing chroma variations. For example, when an image with a brightlylit apple is optimized for rendering on an HDR display, the rendering onan LDR display with reduced dynamic range will typically make the appleappear to stand out less and to appear less clear and duller. This mayby the dynamic range transform be compensated for by making the color ofthe apple more saturated. As another example, the texture variations maybecome less perceptually significant due to the reduced luminancevariations and this may be compensated by increasing the chromavariations of the texture.

In some systems, the video signal may comprise a data field whichincludes dynamic range transform control data and the dynamic rangeprocessor 203 may adapt the dynamic range transform in response to thiscontrol data. This may be used by the content owner/ provider to retainat least some input to or control over the rendering of the providedcontent.

The control data may for example define an operation or parameter of thedynamic range transform that must be applied, may be applied, or whichis recommended to be applied. The control data may furthermore bedifferentiated for different end-user displays. For example, individualcontrol data may be provided for a plurality of possible end-userdisplays, such as one set of data for a 500 nits display, another setfor a 1000 nits display, another set of a 2000 nits display, and yetanother set for a 4000 nits display.

As an example, the content creator may specify which tone mapping shouldbe performed by the dynamic range processor 203 depending on theend-user display characteristics as illustrated in FIG. 11. In theexample, the control data may specify a mapping for each of three areascorresponding to given values of the maximum luminance of the display(x-axis) and the ambient light incident on the display (and thus thereflections from the display—y-axis).

Thus, in the specific example mapping 1 is used for low brightnessdisplays in low ambient light environments. Mapping 1 may simply be aunity mapping, i.e. the received LDR image may be used directly. For ahigh maximum luminance (HDR) display in a relatively dark ambientenvironment (low screen reflections), mapping 2 may be used. Mapping 2may perform a mapping which extends the bright luminances of the LDRimage further while substantially maintaining the intensity for thedarker segments. For a high maximum luminance (HDR) display in arelatively bright ambient environment (substantial screen reflections),mapping 3 may be used. Mapping 3 may perform a more aggressive mappingwhich not only extends the bright luminances of the LDR image but alsobrightens and increases contrast for the darker image areas.

In some scenarios, the control data may specify the boundaries betweenthe mappings with the mappings being predetermined (e.g. standardized orknown at both the content provider side and at the renderer side). Insome scenarios, the control data may further define elements of thedifferent mappings or may indeed specify the mappings precisely, e.g.using a gamma value or specifying a specific transformation function.

In some embodiments, the dynamic range transform control data maydirectly and explicitly specify the dynamic range transform that shouldbe performed to transform the received image to an image with adifferent dynamic range. For example, the control data may specify adirect mapping from the input image values to output image values for arange of target output display white points. The mapping may be providedas a simple parameter allowing the appropriate transform to be realizedby the dynamic range processor 203 or detailed data may be provided suchas a specific look up table or mathematical function.

As a low complexity example, the dynamic range transform may simplyapply a piecewise linear function to the input values of an LDR image togenerate improved HDR values. Indeed, in many scenarios, a simplemapping consisting of two linear relationships as illustrated in FIG. 12can be used. The mapping shows a direct mapping between input pixelvalues and output pixel values (or in some scenarios the mapping mayreflect a (possibly continuous) mapping between input pixel luminancesand output pixel luminances). It will be appreciated that the samemapping may be used to map from an input HDR image to an output LDRimage.

Specifically, for a mapping from LDR to HDR, the approach provides adynamic range transform which maintains the dark areas of an image toremain dark while at the same time allows the substantially increaseddynamic range to be used to provide a much brighter rendering of brightareas, as well as indeed an improved and more lively looking midrange.For a mapping from HDR to LDR, the approach provides a dynamic rangetransform which maintains the dark areas of an image but compresses thebrighter areas to reflect the reduced brightness range of the display.

However, the exact transformation depends on the target display forwhich the image was generated and on the display on which it is to berendered. For example, when rendering an image for a 500 nits display ona 1000 nits display, a relatively modest transformation is required andthe stretching of the bright areas is relatively limited. However, ifthe same image is to be displayed on a 5000 nits display, a much moreextreme transformation is required in order to fully exploit theavailable brightness without brightening the dark areas too much.

Likewise the mapping may depend on the target display for which theoriginal image was generated. For example, if an input image optimizedfor 1000 nits is to be rendered on a 2000 nits display, a relativelymodest transformation is required and the stretching of the bright areasis relatively limited. However, if an image has been optimized for 500nits display and is to be displayed on a 2000 nits display, a much moreextreme transformation is required in order to fully exploit theavailable brightness without brightening the dark areas too much. FIG.13 illustrates how two different mappings may be used for respectively a1000 nits input image (curve 1301, maximum value of 255 corresponding to1000 nits) and a 500 nits input image (curve 1303 maximum value of 255corresponding to 500 nits) for display on a 2000 nits LDR input image(maximum value of 255 corresponding to 2000 nits).

An advantage of such a simple relationship is that the desired tonemapping may be communicated with a very low overhead. Indeed, thecontrol data may specify the knee of the curve, i.e. the point of thetransition between the two linear pieces. Thus, a simply two componentdata value may specify the desired tone mapping to be performed by theimage processing device 103 for different displays. The image processingdevice 103 may further determine suitable values for other maximumluminance values by interpolating between the provided values.

In some implementations, more points may e.g. be provided to define acurve which is still piecewise linear but with more linear intervals.This may allow a more accurate tone mapping and improve the resultingimage quality while only introducing a relatively minor overhead.

In many implementations, the control data may not specify a specifictone mapping that should be performed but rather provide data whichdefines boundaries within which the dynamic range transform/ tonemapping may be freely adapted by the image processing device 103.

For example, rather than specify a specific transition point for thecurves of FIGS. 12 and 13, the control data may define limits for thetransition point (with possibly different limits being provided fordifferent maximum brightness levels). Thus, the image processing device103 may individually determine desired parameters for the dynamic rangetransform such that this can be set to provide the preferred transitionfor the specific display taking into account e.g. the specific userpreferences. However, at the same time the content provider can ensurethat this freedom is restricted to an acceptable range thereby allowingthe content provider to retain some control over how the content isrendered.

Thus, the dynamic range transform control data may include data thatdefines transform parameters that must be applied by the dynamic rangetransform performed by the dynamic range processor 203 and/or whichdefine limits for the transform parameters. The control data may providesuch information for a range of maximum brightness levels therebyallowing adaptation of the dynamic range transform to different end-userdisplays. Furthermore, for maximum brightness levels not explicitlyincluded in the control data, appropriate data values may be generatedfrom the available data values, e.g. by interpolation. For example, if aknee point between two linear pieces is indicated for a 2000 nits and a4000 nits end-user display, a suitable value for a 3000 nits display maybe found by simple interpolation (e.g. by a simple averaging in thespecific example).

It will be appreciated that many different and varied approaches forboth the dynamic range transform and for how to restrict, adapt andcontrol this from the content provider side by additional control datamay be used in different systems depending on the specific preferencesand requirements of the individual application.

Indeed, many different commands or parameter values can be provided inthe control data to generate tone mappings in accordance with thepreferences of the content provider.

For example, in low complexity systems, a simple dynamic range transformmay be applied and the content provider apparatus 101 may simply providea white level and black level for the target display which is then usedby the dynamic range processor 203 to determine the tone mapping toapply. In some systems a tone mapping function (gamma or otherwise) maybe provided as mandatory for mapping at least one range of the inputimage. For example, the control data may specify that darker and/ormidranges must be rendered in accordance with a given mapping whileallowing brighter ranges to be mapped freely by the image processingdevice 103.

In some scenarios, the control data may merely provide a suggestion ofsuitable mapping that can be applied e.g. in the midrange area. In sucha case, the content provider may thus assist the image processing device103 in providing suggested dynamic range transform parameters which havebeen found (e.g. through manual optimization by the content provider) toprovide a high image quality when viewed on a given HDR display. Theimage processing device 103 may advantageously use this but is free tomodify the mapping e.g. to accommodate for individual user preferences.

In many scenarios the mapping is at least partially performed based oncontrol data will represent a relatively low complexity functionalrelationship, such as a gamma mapping, S-curve, combined mapping definedby partial specifications for individual ranges etc. However, in somescenarios more complex mappings may of course be used.

It will also be appreciated that the dynamic range transform may ofteninclude an increase or decrease in the number of bits used to representthe values. For example, an eight bit image may be transformed into a 12or 14 bit image. In such cases, the control data from the contentprovider apparatus 101 may be provided independently of the changedquantization. For example, an 8 bit to 8 bit co-encoded tone mapping(“shape” for grey-sub-distribution) may be defined by the contentprovider apparatus 101 and the image processing device 103 may scalethis mapping to the specific display white brightness by taking intoaccount the transformation to more bits.

In other embodiments or scenarios, the dynamic range transform mayinclude a decrease in the number of bits used to represent the values.For example, a 12 bit image may be transformed into an 8 bit image. Suchscenarios may often occur when a reduction in the dynamic range isprovided by the dynamic range transform, e.g. when converting a 12 bitHDR image to be rendered on an 8 bit input value LDR display.

As mentioned, the control data may provide mandatory or voluntarycontrol data. Indeed, the received data may include one or more fieldsthat indicate whether the tone mapping parameters provided aremandatory, allowed, or suggested.

For example, a suggested tone mapping function may be provided togetherwith an indication of how large a deviation therefrom can be accepted.An image processing device 103 in a standard configuration may thenautomatically apply the suggested mapping. However, the transform may bemodified e.g. to reflect a user's personal preferences. For example, auser input may change the settings of the image processing device 103,e.g. such that the dark areas of an image are rendered brighter thanconsidered ideal by the content provider. E.g. a user may simply press abutton for increasing brightness, and the tone mapping may be changedaccordingly (e.g. the lower linear section of the curves of FIGS. 12 and13 are moved upwards). The user may thus introduce a fine tuning to thetone mapping. However, data of how much fine tuning is acceptable to thecontent provider may be included in the control data thereby restrictingthe dynamic range transform to generate output images that are stillconsidered by the content provider to retain the integrity of the imagebeing provided. The control data may e.g. also specify the effect of theuser interactions, such as e.g. define or limit the change in brightnessthat occurs for each press of the button by a user.

The dynamic range transform accordingly provides a dynamic rangetransform which is intended to provide an image which is appropriate forthe specific end-user display 107 while taking into account the displaycharacteristics of the display for which the input image is generated.Thus, the image processing device 103 generates an output signal that isassociated with a given maximum luminance/ brightness value, i.e. whichis intended for rendering on a display with that white point/ maximumluminance value. In some systems, the white point luminance of thedisplay may not be accurately known to the image processing device 103,and thus the output signal may be generated for an assumed white pointluminance (e.g. entered manually by a user). In other applications (aswill be described later), the display may provide information on thewhite point luminance and the image processing device 103 may adapt thedynamic range transform based on this information.

If the white point luminance for which the output signal is generatedcorresponds exactly or sufficiently closely to the white point luminanceof one of the received images (according to any suitable criterion, suchas a difference the white point luminances being below a threshold), theimage processing device 103 may proceed to use this image directly inthe output image, i.e. the dynamic range transform may simply be a unitymapping. Furthermore, if the output white point luminance does notcorrespond directly to a white point luminance of a received image butdoes match an end-user display white point luminance for which explicitdynamic range transform control data has been provided, this controldata may be used directly to adapt the dynamic range transform. If theoutput white point luminance does not correspond directly with the whitepoint luminance of a received image or with a white point luminance forwhich dynamic range transform control data has been provided, the tonemapping parameters provided by the control data for different whitepoint luminances may be used to adapt the dynamic range transform independence on the output white point luminance. In particular, thedynamic range processor 203 may interpolate between the tone mappingparameters for other white point luminance values to the specific outputwhite point luminance. In many embodiments, a simple linearinterpolation will be sufficient but it will be appreciated that manyother approaches may be used.

Indeed, the control data may for example also provide information on howthe provided tone mapping parameters for different display white pointluminances should be processed to generate tone mapping parameters forthe specific output white point luminance. For example, the control datamay indicate a non-linear interpolation function which must be used togenerate appropriate tone mapping parameters.

It will also be appreciated that the dynamic range transform is notnecessarily constant for different images or even for the same image.

Indeed, in many systems the dynamic range transform control data maycontinuously be updated thereby allowing the dynamic range transformperformed by the dynamic range processor 203 to be adapted to thecurrent characteristics. This may allow different tone mappings to beused for dark images/ scenes than for bright images/scenes. This canprovide improved performance. Indeed, a time variable dynamic rangetransform controlled in response to dynamically updated dynamic rangetransform control data can be used to provide additional control to thecontent provider. For example, the rendering of a dark scene may bedifferent on an HDR display depending on whether the scene is a tensescene intended to provide unease or whether the scene is merely dark inorder to correspond to a nighttime scenario (in the first case the darkscene may be rendered as dark on the HDR display as on an LDR display,and in the second case the dark scene may be rendered somewhat lighterthereby exploiting the additional dynamic range to allow improvedvisually perceptible differentiation in dark areas).

The same considerations may be applied within an image. For example, ascene may correspond to a bright sky over a dark shadowy ground (e.g. abright sky in the upper half of the image and a forest in the lower halfof the image). The two areas may advantageously be mapped differentlywhen mapping from LDR to HDR, and the dynamic range transform controldata may specify the difference in these mappings. Thus, the dynamicrange transform control data may include tone mapping parameters thatchange for different images and/or which depend on the position in theimage.

As a specific example, at least some control data may be associated witha given image area, luminance range, and/or image range.

The dynamic range transform control data may be provided to the imageprocessing device 103 in accordance with any suitable communicationapproach or standard.

In the specific example of the communication between the contentprovider apparatus 101 and the image processing device 103 uses aBlu-ray™ medium. Transmission of control commands for the dynamic rangetransform may be achieved by embedding these parameters values in theBDROM data on the disc. An extension data structure in the playlist file(xxxxx.mpls) can be used for this. This extension data structure willhave a unique and new identification. Legacy BDROM players will beignorant of this new data structure and will simply ignore it. This willguarantee backward compatibility. A possible implementation of thesyntax and semantics of such an LHDR_descriptor is shown below.

Syntax No. of bits Mnemonics LHDR_Descriptor ( ) {Video_Process_descriptor 8 uimsbf DR_Process_descriptor 8 uimsbfLevel_Process_descriptor 8 uimsbf Dynamic_range }

In this example the LHDR descriptor contains three processingdescriptors. These parameters specify the additional processing of thevideo in case the target display category is different from the end-userdisplay category. As an example these parameters can have the followingvalues.

Video_Process_descriptor:

Video/Graphics Video/Graphics processing in case of processing in caseof Target Display = LDR Target Display = HDR Value End-user's Display =HDR End-user's Display = LDR 0x00 No additional processing No additionalprocessing 0x01 Allow limited additional Allow limited additionalprocessing depending on processing depending on DR_Process_descriptorand DR_Process_descriptor and Level_Process_descriptor.Level_Process_descriptor. 0x02 No restrictions on No restrictions onadditional processing additional processing 0x03- reserved reserved 0xFFDR_Process_descriptor:

Video/Graphics Video/Graphics processing in case of processing in caseof Target Display = LDR Target Display =HDR Value End-user's Display =HDR End-user's Display = LDR 0x00 Allow increase of Allow decrease ofdynamic range to 125% dynamic range to 80% 0x01 Allow increase of Allowdecrease of dynamic range to 150% dynamic range to 70% 0x02 Allowincrease of Allow decrease of dynamic range to 200% dynamic range to 50%0x03- reserved reserved 0xFFLevel_Process_descriptor:

Video/Graphics Video/Graphics processing in case of processing in caseof Target Display = LDR Target Display = HDR Value End-user's Display =HDR End-user's Display = LDR 0x00 Allow adaptation of Allow adaptationof level range to 80-125% level range to 80-125% 0x01 Allow increase ofAllow increase of level range to 70-150% level range to 70-150% 0x02Allow increase of Allow increase of level range to 50-200% level rangeto 50-200% 0x03- reserved reserved 0xFF

The previous examples focused on examples wherein the signal receivedfrom the content provider apparatus 101 comprises only one version ofthe image/ video sequence, and specifically where the signal comprisesonly an LDR image/ video sequence.

However, in some systems and implementations, the content providerapparatus 101 may generate an image signal which comprises more than oneversion of the image(s). In such scenarios one image may be tone mappedfor one target display and another image may correspond to the sameoriginal image but tone mapped for a different target display.Specifically, one image may be an LDR image generated for e.g. a 500nits display and another image may be an HDR image generated for e.g. a2000 nits display.

In such an example, the image signal may further comprise a secondtarget display reference, i.e. a target display reference may beprovided for each of the images thereby indicating the displaycharacteristics for which the tone mapping at the encoder side has beenoptimized for the individual images. Specifically, a maximum brightnessand gamma parameter may be provided for each image/ video sequence.

In such systems, the image processing device 103 can be arranged toapply the dynamic range transform in response to the second targetdisplay reference, and specifically by considering both the first andsecond target display references.

The dynamic range transform may not only adapt the specific mapping oroperation that is performed on an image but may also depending on thetarget display references select which image to use as the basis for thetransformation. As a low complexity example, the dynamic range processor203 may select between using the first and second images depending onhow closely the associated target display reference matches the whitepoint luminance for which the output signal is generated. Specifically,the image associated with a white point luminance closest to the desiredoutput white point luminance can be selected. Thus, if an LDR outputimage is generated, the dynamic range transform may be performed fromthe encoded LDR image. However, if an HDR image with higher maximumbrightness than the encoded HDR image is generated, the dynamic rangetransform may be performed on the encoded HDR image.

If an image is to be generated for a maximum brightness between thewhite point luminances of the encoded images (e.g. for a 1000 nitsdisplay), the dynamic range transform may be based on both images. Inparticular, an interpolation between the images may be performed. Suchan interpolation may be linear or non-linear and may be performeddirectly on the encoded images prior to transformation or may be appliedon images after application of the transformation. The weighting of theindividual images can typically depend on how closely they are to thedesired output maximum brightness.

For example, a first transformed image may be generated by applying adynamic range transform to the first encoded image (the LDR image) and asecond transformed image may be generated by applying a dynamic rangetransform to the second transformed image. The first and secondtransformed images are then combined (e.g. summed) to generate theoutput image. The weights of respectively the first and the secondtransformed images are determined by how closely the target displayreferences of respectively the first and second encoded images match thedesired output maximum brightness.

For example, for a 700 nits display, the first transformed image may beweighted much higher than the second transformed image and for a 3000nits display the second transformed image may be weighted significantlyhigher than the first transformed image. For a 2000 nits display the twotransformed images may possibly be weighted equally and the outputvalues may be generated by an averaging of the values for each image.

As another example, the transformation may be performed selectivelybased on the first or second image for different image areas, e.g.depending on image characteristics.

For example, for relatively dark areas the dynamic range transform maybe applied to the LDR image to generate pixel values that are suitablefor a 1000 nits display yet utilize the finer resolution that may beavailable for dark areas for the LDR image corresponding to the HDRimage (e.g. if the same number of bits are used for both images).However, for brighter areas the pixel values may be generated byapplying a dynamic range transform to the HDR image thereby exploitingthat this image will typically have more information in the highbrightness ranges (specifically the information loss due to clipping istypically much less for an HDR image relative to an LDR image).

Thus, when more than one image are received from the content providerapparatus 101 the image processing device 103 may generate the outputimage from one of these images or may combine them when generating anoutput image. The selection and/or combination of the encoded images isbased on the target display reference provided for each image as well ason the maximum brightness for which the output signal is generated.

It will be appreciated that in addition to the combination and/orselection of the individual encoded images, the individual dynamic rangetransforms may also be adjusted and adapted in response to the dynamicrange transform. For example, the previously described approaches may beapplied individually to each dynamic range transform. Similarly, dynamicrange transform control data may be received which can be used to adaptand control each dynamic range transform as previously described. Inaddition, the dynamic range transform control data may containinformation that defines mandatory, optional or preferred/ suggestedparameters for the combination of the processing of the first and secondencoded images.

In some systems, dynamic range transform control data comprisesdifferent transform control data for different image categories.Specifically, different types of images/content may be processeddifferently when performing the dynamic range transform.

For example, different tone mappings may be defined or suggested fordifferent types of video content. For example, a different dynamic rangetransform is defined for a cartoon, a horror film, a football game etc.The received video signal may in such a case provide metadata describingthe content type (or content analysis may be applied locally in theimage processing device 103) and apply the appropriate dynamic rangetransform for the specific content.

As another example, a rendered image may be generated as a combinationof overlaid images with different transforms being provided for thedifferent images. For example, in Blu-ray™ a number of differentpresentation planes are defined (as illustrated in FIG. 5) and differentdynamic range transforms may be applied for the different presentationplanes.

The characteristics of each of these presentation planes are optimizedby the content provider for a specific target display. The viewingexperience for the end-user can be optimized by adapting thecharacteristics of the presentation planes to the end-user display.Typically the optimal adaptation will be different for the differentpresentation planes.

With respect to tone mapping the situation in the present day BDROMsystem is as follows:

-   -   Video tone mapping (global and/or local) is performed in the        studio using a studio monitor.    -   Graphics tone mapping (generally different from Video tone        mapping) is performed in the studio using a studio monitor.    -   OSD tone mapping is performed in the BDROM player.    -   Global and/or local tone mapping is performed in the display on        the combined Video & Graphics signal. This processing cannot be        controlled by the end-user.    -   Global tone mapping is performed in the display on the combined        Video & Graphics signal. This processing depends on, among other        things, the brightness and contrast values set by the end-user.

Improved picture quality is achieved when:

1. Video tone mapping is optimized for the end-user display.

2. Graphics tone mapping is optimized for the end-user display.

3. The system allows for Graphics tone mapping different from Video tonemapping.

4. The System allows for different Graphics tone mapping for differentGraphics components

5. The system allows for Video & Graphics tone mapping depending onVideo characteristics.

Also note that in case that both an LDR and an HDR version of the Videoare present on the disc, the additional tone mapping will depend on twosets of parameters for the target displays: one for the LDR version ofthe video and one for the HDR version of the video.

In another enhanced implementation, the Video and/or Graphics tonemapping varies in time and depend for example on the Video content in ascene. The content provider may send tone mapping instructions to theplayer depending on the characteristics of the Video and Graphicscontent. In another implementation, the player autonomously extracts thecharacteristics of the Video from the Video signal and adapts the Video& Graphics tone mapping depending on these characteristics.

E.g. subtitles may be dimmed for a certain time span, or a certain gammachange may be implemented for an amount of time (and both may becoordinated).

In the following an example of how to provide control commands forGraphics tone mapping for a BDROM is described.

A BDROM graphics stream consists of segments embedded in PES packetsthat are embedded in a transport stream. FIG. 14 illustrates theappropriate data structure.

Synchronization with the main video is done at elementary stream levelusing PTS values in the PES packets. The BDROM graphics segment consistsof a segment descriptor and the segment data. The segment descriptorcontains the type of the segment and the length.

The following table shows some types of segments defined in the Blu-rayDisc standard:

Value Segment  0x00- reserved 0x13 0x14 Palette Definition Segment 0x15Object Definition Segment 0x16 Presentation Composition Segment 0x17Window Definition Segment 0x18 Interactive Composition Segment  0x19-reserved 0x7F 0x80 End of Display Set Segment  0x81- Used by HDMV Text0x82 subtitle streams 0x83 LHDR_Processing_Definition_Segment  0x84-reserved 0xFF

In the existing specification, values 0x83 to 0xFF are reserved.Therefore a new segment type is defined using for example value 0x83 toindicate a segment that contains the LHDR_Processing_definition segment.In general, the LHDR_Processing_definition segment defines the way thegraphics decoder processes the graphics in case of the target displaybeing different from the end-user display.

The following table shows an example of a possible structure of theLHDR_Processing_definition segment:

Syntax No. of bits Mnemonics LHDR_Processing_definition segment ( ) { segment_descriptor( ) 8 uimsbf  Pup-up_process_descriptor 8 uimsbf Subtitle_process_descriptor 8 uimsbf  Number_of_HDR_Palettes 8 uimsbf for (i=0; i<Number_of_HDR_Palettes; i++) {   palette_id 8 uimsbf  palette_version_number 8 uimsbf   Number_of_entries 8 uimsbf   for(i=0; i< Number_of_entries; i++) {    palette_entry( ){ Palette_entry_id 8 uimsbf     Y_value 12 uimsbf     Cr_value 12 uimsbf    Cb-value 12 uimsbf     T_value 12 uimsbf    }   }  }

In this example, the LHDR_Processing_definition segment contains twoprocessing descriptors: Pop-up_process_descriptor andSubtitle_process_descriptor. The segment may also contain palettes to beused in case the target display category is different from the end-userdisplay category. The LHDR palette contains the same number of entriesas the original palette but the entries are optimized for the otherdisplay category.

The parameter Pop-up_process_descriptor specifies the additionalprocessing of the Pop-up graphics in case target display category isdifferent from the end-user display category.

As an example this parameter can have the following values.

-   -   Pop-up_process_descriptor=0x00: No additional processing.    -   Pop-up_process_descriptor=0x01 to 0x03: set minimum transparency        value.    -   Pop-up_process_descriptor=0x04: the graphics processor uses        palettes defined in the LHDR_Processing_definition segment.    -   Pop-up_process_descriptor=0x05: No restrictions on additional        processing.

The parameter Subtitle_process_descriptor specifies the additionalprocessing of Subtitle graphics in case the target display category isdifferent from the end-user display category.

As an example this parameter can have the following values.

-   -   Subtitle_process_descriptor=0x00: No additional processing.    -   Pop-up_process_descriptor=0x01 to 0x03: Adapt luma value.    -   Subtitle_process_descriptor=0x04: the graphics processor uses        palettes defined in the LHDR_Processing_definition segment.    -   Subtitle_process_descriptor=0x05: No restrictions on additional        processing.

Specific examples of syntaxes for the Pop-up_process_descriptor and theSubtitle_process_descriptor are provided in the following tables:

Graphics processing in case of Graphics processing in case of TargetDisplay = LDR Target Display = HDR Value End-user's Display = HDREnd-user's Display = LDR 0x00 No additional processing No additionalprocessing 0x01 Set T_value >= 128 No additional processing 0x02 SetT_value >= 192 No additional processing 0x03 Set T_value >= 222 Noadditional processing 0x04 Use LHDR palettes Use LHDR palettes 0x05 Norestrictions No restrictions  0x06- reserved reserved 0xFF

Graphics processing in case of Graphics processing in case of TargetDisplay = LDR Target Display = HDR Value End-user's Display = HDREnd-user's Display = LDR 0x00 No special processing No specialprocessing 0x01 Luma:=Luma/5 Luma:=Luma*5 0x02 Luma:=Luma/3 Luma:=Luma*30x03 Luma:=Luma/2 Luma:=Luma*2 0x04 Use LHDR palettes Use LHDR palettes0x05 No restrictions No restrictions  0x06- reserved reserved 0xFF

Specific examples of differentiated tone mapping depending on displaycharacteristics are illustrated in FIGS. 15 and 16. In these examples,the original content features HDR video content and subtitles. Tonemapping for the video is the same as in the example of FIG. 6.

The Graphics features white sub-title characters with a black border.The original histogram shows a peak in the low-luma range and anotherpeak in the high luma range. This histogram for the subtitle content isvery suitable for a LDR display as it will result in bright legible texton the display. However, on a HDR display these characters would be toobright causing annoyance, halo and glare. For that reason, the tonemapping for the sub-title graphics will be adapted as depicted in FIG.16.

In the previous examples, the image processing device 103 has generatedan output image to correspond to a desired maximum brightness, i.e.intended for presentation on a display with a given dynamic range/whitepoint luminance. The output signal may specifically be generated tocorrespond to a user setting which indicates a desired maximum/whitepoint luminance, or may simply assume a given dynamic range for thedisplay 107.

In some systems the image processing device 103 may comprise a dynamicrange processor 203 which is arranged to adapt its processing independence on data received from the display 107 indicating a luminancecharacteristic of the display 107.

An example of such an image processing device 103 is illustrated in FIG.17. The image processing device 103 corresponds to that of FIG. 1 but inthis example the image processing device 103 also comprises a displayreceiver 1701 which receives a data signal from the display 107. Thedata signal comprises a data field which comprises a display dynamicrange indication for the display 107. The display dynamic rangeindication comprises at least one luminance specification indicative ofa luminance property of the display. Specifically the luminancespecification may include a specification of a maximum brightness, i.e.of a maximum/white point luminance for the display. Specifically, thedisplay dynamic range indication can define whether the display is anHDR or LDR display and may in particular indicate the maximum lightoutput in nits. Thus, the display dynamic range indication can define ifthe display is a 500 nits, 1000 nits, 2000 nits, 4000 nits etc display.

The display receiver 1701 of the image processing device 103 is coupledto the dynamic range processor 203 which is fed the display dynamicrange indication. The dynamic range processor 203 can accordinglygenerate an output signal which directly corresponds to the specificdisplay rather than to generate the output signal for an assumed ormanually set white point luminance.

The dynamic range processor 203 may accordingly adapt the dynamic rangetransform in response to the received display dynamic range indication.For example, the received encoded image may be an LDR image and it maybe assumed that this image hasbeen optimized for a 500 nits display. Ifthe display dynamic range indication indicates that the display isindeed a 500 nits display, the image processing device 103 may use theencoded image directly. However, if the display dynamic range indicationindicates that the display is a 1000 nits display, a first dynamictransform may be applied. If the display dynamic range indicationindicates that the display 107 is a 2000 nits display, a differenttransform may be applied, etc. Similarly, if the received image is a2000 nits optimized image, the image processing device 103 may use thisimage directly if the display dynamic range indication indicates thatthe display is a 2000 nits display. However, if the display dynamicrange indication indicates that the display is a 1000 nits or a 500 nitsdisplay, the image processing device 103 may perform the appropriatedynamic range transform to reduce the dynamic range.

For example, referring to FIG. 18, two different transformations may bedefined for respectively a 1000 nits display and for a 4000 nitsdisplay, and with a third one-to-one mapping being defined for a 500nits display. In FIG. 1, the mapping for the 500 nits display isindicated by curve 1801, the mapping for the 1000 nits display isindicated by curve 1803, the mapping for the 4000 nits display isindicated by curve 1805. Thus, in the example, the received encodedimage is assumed to be a 500 nits image and this is automaticallyconverted into an image suitable for the specific display. Thus, theimage processing device 103 can automatically adapt and generate anoptimized image for the specific display to which it is connected. Inparticular, the image processing device 103 can automatically adapt towhether the display is an HDR or LDR display, and can further adapt tothe specific white luminance of the display.

It will be appreciated that the inverse mappings may be used whenmapping from a higher dynamic range to a lower dynamic range.

If the display has a white luminance corresponding to one of the threecurves of FIG. 18, the corresponding mapping may be applied to theencoded image. If the display has a different luminance value, acombination of the transformations may be used.

Thus, the dynamic range processor 203 may select an appropriate dynamicrange transform depending on the display dynamic range indication. As alow complexity example, the dynamic range processor 203 may selectbetween using the curves depending on how closely the associated whitepoint luminance matches the white point luminance indicated by thedisplay dynamic range indication. Specifically, the mapping that isassociated with a white point luminance closest to the desired whitepoint luminance indicated in the display dynamic range indication can beselected. Thus, if an LDR output image is generated, the dynamic rangetransform may be performed using curve 1801. If a relatively low whitepoint luminance HDR image is generated, the mapping of curve 1803 isused. However, if high white point luminance HDR image is generated,curve 1805 is used.

If an image is to be generated for a white luminance in-between thedynamic range transforms for the two HDR settings (e.g. for a 2000 nitsdisplay), both mappings 1803, 1805 may be used. In particular, aninterpolation between the transformed images for the two mappings may beperformed. Such an interpolation may be linear or non-linear. Theweighting of the individual transformed images can typically depend onhow closely they are to the desired output maximum brightness.

For example, a first transformed image may be performed by applying afirst mapping 1803 to the encoded image (the LDR image) and a secondtransformed image may be performed by applying a second mapping to theencoded image. The first and second transformed images are then combined(e.g. summed) to generate the output image. The weights of respectivelythe first and the second transformed images are determined by howclosely the white luminance associated with the different mappings matchthe display white luminance indicated in the display dynamic rangeindication.

For example, for a 1500 nits display, the first transformed image may beweighted much higher than the second transformed image and for a 3500nits display the second transformed image may be weighted significantlyhigher than the first transformed image.

In some embodiments, the dynamic range processor (203) may be arrangedto select between generating the output image as the received encodedimage and generating the output image as a transformed image of thereceived encoded image in response to the display dynamic rangeindication.

Specifically, if the white point luminance indicated by the displaydynamic range indication is sufficiently close to the white pointluminance indicated or assumed for the received image, the dynamic rangetransform may simply consist in not performing any processing on thereceived image, i.e. the input image may simply be used as the outputimage. However, if the white point luminance indicated by the displaydynamic range indication is different to the white point luminanceassumed or indicated for the received image, the dynamic range transformmay modify the received encoded image in accordance with a suitablemapping of the input image pixels to output image pixels. In such cases,the mapping may be adapted depending on the received indication of thewhite point luminance of the end user display. In other examples, one ormore predetermined mappings may be used.

For example, the image processing device 103 may include a predeterminedfirst mapping which has been determined to provide a suitable outputimage for a doubling in the white point level and a predetermined secondmapping which has been determined to provide a suitable output image fora halving in the white point level. In such an example, the imageprocessing device 103 may select between the first mapping, the secondmapping, and a unity mapping dependent on the white point luminance ofthe received image (e.g. as indicated by the target display reference)and the white point luminance for the end user display as indicated bythe display dynamic range indication. The image processing device 103may specifically select the mapping which most closely corresponds tothe ratio between the white point luminances of the input image and theend-user display.

For example, if an input image is received with a target displayreference indicating that it has been optimized for a 1000 nits displayand the end-user display is a 2000 nits display, the image processingdevice 103 will select the first mapping. If instead, the displaydynamic range indication indicates that the end-user display is a 1000nits display, the image processing device 103 will select the unitymapping (i.e. use the input image directly). If the dynamic rangeindication indicates that the end-user display is a 500 nits display,the image processing device 103 will select the second mapping.

If in-between values for the white point luminance of the end-userdisplay are received, the image processing device 103 may select themapping closest to the ratio between the white point luminances, or maye.g. interpolate between the mappings.

In the example of FIG. 2, the image processing device 103 is arranged toperform a dynamic range transform based on a target display referencereceived from the content provider apparatus 101 but without anyspecific information or knowledge of the specific display 107 (i.e. itmay simply generate the output image to be optimized for a given dynamicrange/ white point but without explicitly knowing if the connecteddisplay 107 has that value). Thus, an assumed or reference white pointluminance may be used. In the example of FIG. 17, the image processingdevice 103 may perform a dynamic range transform based on a displaydynamic range indication received from the display 107 but without anyspecific information or knowledge of the specific dynamic range andwhite point luminance that the received encoded image has been generatedfor (i.e. it may simply generate the output image based on given dynamicrange/ white point luminance for the received encoded image but withoutexplicitly knowing if the image has actually been generated for such arange and luminance). Thus, an assumed or reference white pointluminance for the encoded image may be used. However, it will beappreciated that in many implementations the image processing device 103may be arranged to perform the dynamic range transform in response toboth information received from the content provider side and from theend-user display. FIG. 19 shows an example of an image processing device103 which comprises a dynamic range processor 203 arranged to perform adynamic range transform in response to both the target display referenceand the display dynamic range indication. It will also be appreciatedthat the comments and descriptions provided for the independentapproaches of FIGS. 2 and 17 apply equally (mutatis mutandis) to thesystem of FIG. 19.

The approaches may be particularly advantageous in inhomogeneous contentdistribution systems such as e.g. what is increasingly perceived forfuture television systems. Indeed the (peak) brightness of displays iscurrently rapidly increasing and in the near future, displays with awide variety of (peak) brightness are expected to coexist in the market.Since the display brightness (and typically the electro-optical transferfunction that specifies how a display converts input pixel (color)driving values into light values which then provides a particularpsychovisual impression to the viewer) is no longer known at the contentgeneration side (and which is furthermore generally different from thereference monitor for which the content was intended/graded), it becomeschallenging to provide the best/optimal picture quality on the display.

Therefore, in the system of FIG. 1 the display 107 (or sink device) cansend information about its brightness capabilities (peak brightness,grey(color) rendering transfer function, or other grey renderingproperties over its HDR range, like a particular electro-opticaltransfer function etc.) back to the image processing device 103.

In the specific example the image processing device 103 is a BDROMplayer connected to a display by means of a HDMI interface, and thus thedisplay dynamic range indication may be communicated from the display tothe image processing device 103 via an HDMI interface. Thus, the displaydynamic range indication can specifically be communicated as part of theEDID information which can be signaled over HDMI from the display 107 tothe image processing device 103. However, it will be appreciated thatthe approach can be applied to many other video/graphics generatingdevices like DVB receivers, ATSC receivers, Personal computers, tablets,smart phones and game consoles etc. It will also be appreciated thatmany other wired and wireless interfaces can be used such as DisplayPort, USB, Ethernet and WIFI etc.

The image processing device 103 can then select e.g. one of differentversions of the content/signal depending on e.g. the display brightness.For example, if the signal from the content provider apparatus 101comprises both an LDR and HDR image, the image processing device 103 canselect between these based on whether the display dynamic rangeindication is indicative of the display being an LDR display or an HDRdisplay. As another example, the image processing device 103 caninterpolate/mix different brightness versions of the content to derive anew signal that is approximately optimal for the display brightness. Asanother example, it can adapt the mapping from the encoded image to theoutput image.

It will be appreciated that in different implementations differentparameters and information may be provided in the display dynamic rangeindication. In particular, it is noted that the previously providedcomments and descriptions for the target display reference may applyequally to the display dynamic range indication. Thus, the parametersand information communicated from the display 107 to the imageprocessing device 103 may be as those described for communication ofinformation on the target display from the content provider apparatus101 to the image processing device 103.

Specifically, the display can communicate a maximum luminance/whitepoint luminance for the display and this may be used by the dynamicrange processor 203 to adapt the output signal as previously described.

In some embodiments, the display dynamic range indication mayalternatively or additionally include a black point luminance for thedisplay 107. The black point luminance may typically indicate aluminance corresponding to drive values corresponding to the darkestpixel value. The intrinsic black point luminance for a display may forsome displays correspond to practically no light output. However, formany displays the darkest setting of e.g. the LCD elements still resultin some light output from the display resulting in black image areasbeing perceived lighter and greyish rather than deep black. For suchdisplays, the information of the black point luminance can be used bythe dynamic range processor 203 to perform a tone mapping where e.g. allblack levels below the black point luminance of the display will beconverted to the deepest dark pixel value (or e.g. using a more gradualtransition). In some scenarios the black point luminance may include acontribution from ambient light. For example, the black point luminancemay reflect the amount of light being reflected from the display.

In addition, the display dynamic range indication may for many displaysinclude more information characterizing the OETF of the display.Specifically, as previously mentioned, the display can include the whitepoint luminance and/or the black point luminance. In many systems, thedisplay dynamic range indication may also include more details about theOETF of the display at intervening light outputs. Specifically, thedisplay dynamic range indication can include a gamma of the OETF for thedisplay.

The dynamic range processor 203 can then use information of the thisOETF to adapt the specific dynamic range transform to provide thedesired performance and in particular, the conversion to an HDR imagemay reflect not only that a brighter light output is possible but mayalso take into consideration exactly how the relationship between thedrive values should be generated to provide the desired light output inthe increased brightness range. Similarly, the conversion to an LDRimage may reflect not only that a less bright light output is availablebut may also take into consideration exactly how the relationshipbetween the drive values should be generated to provide the desiredlight output in the reduced brightness range.

The display dynamic range indication may thus specifically provideinformation that informs the dynamic range processor 203 of how itshould map input values corresponding to one dynamic range to outputvalues corresponding to another and typically larger dynamic range. Thedynamic range processor 203 can take this into consideration and can forexample compensate for any variations or non-linearities in therendering by the display 107.

It will be appreciated that many different dynamic range transforms arepossible and that many different ways of adapting such dynamic rangetransforms based on the display dynamic range indication may be used.Indeed, it will be appreciated that most of the comments provided forthe dynamic range transform based on the target display reference fromthe content provider apparatus 101 are equally appropriate (mutatismutandis) to the dynamic range transform based on information of theluminance characteristics of the end-user display.

As a low complexity example, the dynamic range transform may simplyapply a piecewise linear function to the input values of an LDR image togenerate improved HDR values (or to the input values of an HDR image togenerate improved LDR values). Indeed, in many scenarios, a simplemapping consisting of two linear relationships as illustrated in FIG. 20can be used. The mapping shows a direct mapping between input pixelvalues and output pixel values (or in some scenarios the mapping mayreflect a (possibly continuous) mapping between input pixel luminancesand output pixel luminances).

Specifically, the approach provides a dynamic range transform whichmaintains the dark areas of an image to remain dark while at the sametime allows the substantially increased dynamic range to be used toprovide a much brighter rendering of bright areas, as well as indeed animproved and more lively looking midrange. However, the exacttransformation depends on the display on which it is to be rendered. Forexample, when rendering an image for a 500 nits display on a 1000 nitsdisplay, a relatively modest transformation is required and thestretching of the bright areas is relatively limited. However, if thesame image is to be displayed on a 5000 nits display, a much moreextreme transformation is required in order to fully exploit theavailable brightness without brightening the dark areas too much. FIG.20 illustrates how two different mappings may be used for respectively a1000 nits display (curve 2001, maximum value of 255 corresponding to1000 nits) and a 5000 nits display (curve 2003 maximum value of 255corresponding to 5000 nits) for a 500 nits LDR input image (maximumvalue of 255 corresponding to 500 nits). The image processing device 103may further determine suitable values for other maximum luminances byinterpolating between the provided values. In some implementations, morepoints may be used to define a curve which is still piecewise linear butwith more linear intervals.

It will be appreciated that the same mappings can be used when mappingfrom an HDR input image to an LDR output image.

In some embodiments, the dynamic range transform may comprise or consistin a gamut transform which may be dependent on the received displaydynamic range indication. Thus, in some embodiments, the dynamic rangeprocessor 203 may modify chromaticities of the rendered image dependingon the display dynamic range indication. For example, when a receivedHDR image is rendered on an LDR display the compression may result in ablander image with fewer variations and gradations in individual imageobjects. The dynamic range transform may compensate for such reductionsby increasing chroma variations. For example, when an image with abrightly lit apple is optimized for rendering on an HDR display, therendering on an LDR display with reduced dynamic range will typicallymake the apple appear to stand out less and appear less clear and moredull. This may by the dynamic range transform be compensated by makingthe color of the apple more saturated. As another example, the texturevariations may become less perceptually significant due to the reducedluminance variations and this may be compensated by increasing thechroma variations of the texture.

The display dynamic range indication may in some examples or scenariosprovide generic information for the display, such as the standardmanufacturing parameters, the default EOTF etc. In some examples andscenarios, the display dynamic range indication may further reflect thespecific processing performed in the display and may specificallyreflect user settings. Thus, in such examples, the display dynamic rangeindication does not merely provide fixed and unchanging information thatdepends only on the display but rather provides a time varying functionthat may reflect the specific operation of the display.

For example, the display may be able to operate in different image modeswith different rendering characteristics. For example, in a “vivid”display mode, the display may render images with the bright areasbrighter than normal, in a “mute” display mode the display may renderthe images with the bright areas darker than normal etc. The informationon the current mode, e.g. the specific gamma for this mode, can bereported to the image processing device 103 as part of the displaydynamic range indication thereby allowing the image processing device103 to adapt the dynamic range transform to reflect the renderingcharacteristics. The image processing device 103 may for exampleoverride the display setting by compensating for this or may optimizethe transform to maintain the specific setting.

The display dynamic range indication may also reflect other processingsettings for the display. For example, clipping levels, backlight powersettings, color scheme mappings etc may be communicated to the imageprocessing device 103 where they can be used by the dynamic rangeprocessor 203 to adapt the dynamic range transform.

FIG. 21 illustrates an example of elements of the display 107 where thedisplay provides a display dynamic range indication to the imageprocessing device 103.

In the example, the display comprises a receiver 2101 which receives theimage signal output from the image processing device 103. The receivedimage signal is coupled to a driver 2103 which is further coupled to adisplay panel 2105 which renders the image. The display panel may forexample be an LCD or plasma display panel as will be known to theskilled person.

The driver 2103 is arranged to drive the display panel 2105 such that itrenders the encoded image. In some embodiments, the driver 2103 mayperform advanced and possibly adaptive signal processing algorithmsincluding tone mapping, color grading etc. In other embodiments, thedriver 2103 may be relatively low complexity and may e.g. merely performa standard mapping from the input signal values to drive values for thepixel elements of the display panel 2105.

In the system, the display 107 furthermore comprises a transmitter 2107which is arranged to transmit a data signal to the image processingdevice 103. The data signal may for example for a HDMI connection becommunicated in a DDC channel using the E-EDID structure as will bedescribed later.

The transmitter 2107 generates the data signal to include the displaydynamic range indication for the display (107). Thus, specifically thetransmitter 2107 which indicates e.g. the white point luminance andoptionally the EOTF of the display. For example, a data value providingan index between a number of predetermined white point luminances orEOTFs may be generated and transmitted.

In some low complexity embodiments, e.g. the white point luminance maybe a fixed value stored in the transmitter 2107 which merelycommunicates this standard value. In more complex values, the displaydynamic range indication may be determined to reflect dynamicallyvarying and/or adapted values. For example, the driver 2103 may bearranged to operate in different display modes, and the display dynamicrange indication may be adapted accordingly. As another example, theuser setting of e.g. a brightness level for the display may be reflectedby the display dynamic range indication generated and transmitted by thetransmitter 2107.

As mentioned previously, the display dynamic range indication maycomprise an ambient light measure and the dynamic range processor may bearranged to adapt the dynamic range transform in response to the ambientlight measure. The ambient light measure may be provided as explicit andseparate data or may be reflected in other parameters. For example, theambient light measure may be reflected in the black point luminancewhich may include a contribution corresponding to light reflections fromthe display.

In many scenarios the display may include a light detector positioned atthe front of the display. This light detector may detect the generalambient light level or may specifically measure light reaching thedisplay from a given directly likely to be reflected back towards aviewer. Based on this light detection, the display can thus generate anambient light indication which reflects e.g. the ambient light level ofthe viewing environment in general or e.g. which specifically reflectsan estimate of the reflected light from the screen. The display 107 canreport this value to the image processing device 103, either as anindividual value or e.g. by calculating the effective black luminancelevel to reflect the amount of light reflections.

The dynamic range processor 203 can then adapt the dynamic rangetransform accordingly. For example, when the ambient light level ishigh, more use of the additional bright levels of an HDR display can beused more aggressively to generate a bright looking image with a highcontrast. For example, the average light output may be set relativelyhigh and even midrange luminances may be pushed towards the HDR range.Bright areas may be rendered using the full HDR range and even darkareas would typically be rendered at relatively high light levels.However, the increased dynamic range of an HDR image allows for such arelatively bright image to still exhibit large luminance variations andthus to still have a high contrast.

Thus, the HDR capabilities of the display are used to generate an imagethat provides images which are perceived to be bright and have highcontrast even when viewed e.g. in bright daylight. Such an image wouldtypically not be appropriate in a dark room as it would be overpoweringand appear far too bright. Thus, in a dark environment, the dynamicrange transform would perform a much more conservative LDR to HDRtransform which e.g. maintains the same LDR light output for dark andmidrange values and only increases the brightness for the brighterareas.

The approach may allow the image processing device 103 to automaticallyadapt the LDR to HDR dynamic range transform (or e.g. an HDR to HDRdynamic range transform) to match the specific viewing environment ofthe display. This is furthermore possible without requiring the imageprocessing device 103 to make any measurements of or indeed even to bepositioned in or near this environment.

The ambient light indication may typically be optional and thus theimage processing device 103 may use it if available and otherwise justperform a default dynamic range transform for the specificcharacteristics (e.g. OETF of the display).

The optional extension information provided by the display about itsviewing environment (especially surrounding light) is thus used by theimage processing device 103 to execute more complicated image/videooptimizing transforms for presenting optimal image/video to the displaywhere the optimization can include not only characteristics of thedisplay but also of the viewing environment.

Thus, further optimizations can be performed when information isprovided by the display about the viewing environment. The display willtypically periodically measure the surrounding light and sendinformation (e.g. brightness and color in the form of three parameters:XYZ) about this to the image processing device 103. This information maytypically not be provided as part of EDID data or any other data typeprimarily used for one-time communication of information. Rather, it maybe communicated e.g. in a separate channel, such as e.g. using HDMI-CEC.This periodic measurement and update may e.g. result in that if the usere.g. switches off light in the vicinity of the display, the imageprocessing device 103 can automatically adapt the processing to provideimages more suitable for the darker viewing situation, e.g. by applyingdifferent color/luminance mappings.

An example of a set of relevant parameters that may be reported by theend-user display in the display dynamic range indication includes:

-   -   The absolute maximum luminance (white point luminance) of the        end-user display.    -   Gamma of the end-user display—factory setting.

The absolute maximum luminance of the end-user display might for examplebe defined for typical display settings, factory default settings orsettings producing the highest brightness.

Another example of a set of relevant parameters that may be reported bythe end-user display in the display dynamic range indication includes:

-   -   Maximum luminance of the end-user display for the current        settings of brightness, contrast. etc.    -   Gamma of the end-user display—current settings.

The first set of parameters is time independent whereas the second setvaries in time as it depends on user settings. Application of one or theother set has consequences for the behavior of the system and the userexperience, and it will be appreciated that the specific set ofparameters used in a specific system depends on the preferences andrequirements of the system. Indeed, the parameters can be mixed betweenthe two sets, and e.g. the factory default settings can be provided atswitch-on, with the user setting dependent parameters being reportedperiodically thereafter.

It is also appreciated that the specific parameter sets may characterizean EOTF for the display which is either the factory default EOTF or thespecific current user setting dependent EOTF. Thus, the parameters canprovide information on the mapping between drive values and a luminanceoutput of the display which allows the image processing device 103 togenerate the drive values that will result in the desired output image.It will be appreciated that in other implementations other parametersmay be used to characterize part of or the entire mapping between drivevalues and light output for the display.

It will be appreciated that many different approaches can be used forcommunicating the display dynamic range indication from the display tothe image processing device 103.

For example, for parameters of the display that are independent of usersettings and do not vary over time, the communication may for an HDMIconnection be effectively transferred in a DDC channel using the E-EDIDstructure.

In a low complexity approach, a set of categories may be defined forend-user displays with each category defining ranges of the relevantparameters. In such an approach only a category identification code forthe end-user display needs to be transmitted.

A specific example of a communication of display dynamic rangeindication data in an E-EDID format will be described.

In the specific example, the first 128 bytes of the E-EDID shall containan EDID 1.3 structure (base EDID block).

For the display dynamic range indication parameters, a new DisplayDescriptor Block in the E-EDID data structure may be defined. As currentdevices are ignorant of such a new Display Descriptor Block, they willmerely ignore it thereby providing backwards compatibility. A possibleformat of this “Luminance behavior” descriptor is listed the tablebelow.

Byte # # of bytes Values Description 0, 1 2 00 h Indicates that this 18byte descriptor is a display descriptor 2 1 00 h Reserved 3 1 F6 hDisplay Descriptor Tag number indicating that this is a Luminancedescriptor. 4 1 00 h Reserved 5 1 Peak_Luminance 6-8 3 transfer curve(optional; e.g. alpha, beta, offset)

Peak_Luminance is a parameter with a value between 0 and 255 thatindicates the peak luminance of the display according to:

-   -   display peak luminance (cd/m2)=50×Peak Luminance, thereby        covering a range of 0 to 255*50=12750 cd/m2 or 255*100

The transfer curve may be a gamma curve (as in ITU601, ITU709, etc.) butallowing for a much higher gamma (up to 10). Or a different transfer (orlog) curve parameter may in some scenarios be more appropriate. Forexample, instead of the gamma function:

x^(γ)

a power function:

α^(βx)−Δ

could be used where the parameters α,β and Δ may be set to provide thedesired characterization.

The additional information can thus be used by the image processingdevice 103 to make more advanced decisions for determining differentvideo and graphics (or multi-image component) grey levels, like e.g.global processing such as gamma-based modifications. Having moreinformation, such as on how the display will gamma-remap all greyvalues, the dynamic range processor 203 can make much smarter decisionsfor the final look of video and secondary images (and how they mayoverlap in luminance, depending on also e.g. geometrical properties likehow big the subregions are etc.).

In the previous examples, the display 107 provides a display dynamicrange indication which informs the image processing device 103 of howthe display will display an incoming display signal. Specifically, thedisplay dynamic range indication can indicate the mapping between drivevalues and light output that is applied by the display. Thus, in theseexamples the display dynamic range indication informs the imageprocessing device 103 of the available dynamic range and how this ispresented, and the image processing device 103 is free to adapt thedynamic range transform as it sees fit.

However, in some systems the display may also be able to exert somecontrol over the dynamic range transform performed by the imageprocessing device 103. Specifically, the display dynamic rangeindication can comprise dynamic range transform control data, and thedynamic range processor 203 can be arranged to perform the dynamic rangetransform in response to this dynamic range transform control data.

The control data may for example define an operation or parameter of thedynamic range transform that must be applied, may be applied, or whichis recommended to be applied. The control data may furthermore bedifferentiated for different characteristics of the image to be encoded.For example, individual control data may be provided for a plurality ofpossible initial images, such as one set for a 500 nits LDR image,another for a 1000 nits encoded image etc.

As an example, the display may specify which tone mapping should beperformed by the dynamic range processor 203 depending on the dynamicrange of the received image. For example, for a 2000 nits display, thecontrol data may specify one mapping that should be used when mappingfrom a 500 nits LDR image, and another mapping that should be used whenmapping from 1000 nits image etc.

In some scenarios, the control data may specify the boundaries betweenthe mappings with the mappings being predetermined within each interval(e.g. standardized or known at both the content provider side and at therenderer side). In some scenarios, the control data may further defineelements of the different mappings or may indeed specify the mappingsprecisely, e.g. using a gamma value or specifying a specifictransformation function.

In some embodiments, the dynamic range transform control data maydirectly and explicitly specify the dynamic range transform that shouldbe performed to transform the received image to an image with a dynamicrange corresponding to the dynamic range of the display. For example,the control data may specify a direct mapping from input image values tooutput image values for a range of received image white points. Themapping may be provided as a simple parameter allowing the appropriatetransform to be realized by the dynamic range processor 203 or detaileddata may be provided such as a specific look up table or mathematicalfunction.

As a low complexity example, the dynamic range transform may simplyapply piecewise linear function to the input values of an LDR image togenerate improved HDR values (or to the input values of an HDR image togenerate improved LDR values). Indeed, in many scenarios, a simplemapping consisting of two linear relationships as illustrated in FIG. 20can be used.

Specifically, as previously described, such an approach can provide adynamic range transform which maintains the dark areas of an image toremain dark while at the same time allows the substantially increaseddynamic range to be used to provide a much brighter rendering of brightareas, as well as indeed an improved and more lively looking midrange.However, the exact transformation depends on the dynamic range of thereceived image as well as on the dynamic range of the end targetdisplay. In some systems, the display may thus specify a tone mapping tobe performed by the image processing device 103 simply be communicatingthe coordinates of the knee of the function (i.e. of the intersectionbetween the linear elements of the mapping).

An advantage of such a simple relationship is that the desired tonemapping may be communicated with a very low overhead. Indeed, a simplytwo component data value may specify the desired tone mapping to beperformed by the image processing device 103 for different displays.Different coordinates of the “knee” point may be communicated fordifferent input images and the image processing device 103 may determinesuitable values for other input images by interpolating between theprovided values.

It will be appreciated that most of the comments provided with respectto provision of dynamic range transform control data from the contentprovider apparatus 101 apply equally well (mutatis mutandis) to dynamicrange transform control data received from the display 107.

Thus, in some scenarios the display 107 may be in control of the dynamicrange transform performed by the image processing device 103. Anadvantage of such an approach is that it may e.g. allow a user tocontrol the desired rendered image by controlling the display andwithout any requirement for providing user inputs or settings to theimage processing device 103. This may be particularly advantageous inscenarios where a plurality of image processing devices are used withthe same display, and in particular it may assist in providinghomogeneity between images from different image processing devices.

In many implementations, the control data from the display 107 may notspecify a specific tone mapping that should be performed but ratherprovide data which defines boundaries within which the dynamic rangetransform/ tone mapping may be freely adapted by the image processingdevice 103.

For example, rather than specify a specific transition point for thecurve of FIG. 20, the control data may define limits for the transitionpoint (with possibly different limits being provided for differentmaximum brightness levels). Thus, the image processing device 103 mayindividually determine desired parameters for the dynamic rangetransform such that this can be set to provide the preferred transitionfor the specific display taking into account e.g. the specific userpreferences. However, at the same time display can restrict this freedomto an acceptable level.

Thus, the dynamic range transform control data may include data thatdefines transform parameters that must be applied by the dynamic rangetransform performed by the dynamic range processor 203 and/or whichdefine limits for the transform parameters. The control data may providesuch information for a range of input image dynamic ranges therebyallowing adaptation of the dynamic range transform to different receivedimages. Furthermore, for input images with dynamic ranges not explicitlyincluded in the control data, appropriate data values may be generatedfrom the available data values, e.g. by interpolation. For example, if aknee point between two linear pieces is indicated for a 500 nits and a2000 nits input image, a suitable value for a 1000 nits input image maybe found by simple interpolation (e.g. by a simple averaging in thespecific example).

It will be appreciated that many different and varied approaches forboth the dynamic range transform and for how to restrict, adapt andcontrol this from the display side by additional control data may beused in different systems depending on the specific preferences andrequirements of the individual application.

In some scenarios, the control data may merely provide a suggestion ofsuitable mapping that can be applied e.g. in the midrange area. In sucha case, the display manufacturer may accordingly assist the imageprocessing device 103 in providing suggested dynamic range transformparameters that have been found (e.g. through manual optimization by thedisplay manufacturer) to provide a high image quality when viewed on thespecific display. The image processing device 103 may advantageously usethis but is free to modify the mapping e.g. to accommodate forindividual user preferences.

In many scenarios the mapping which is at least partially performed onthe basis of the control data will represent a relatively low complexityfunctional relationship, such as a gamma mapping, S-curve, combinedmapping defined by partial specifications for individual ranges etc.However, in some scenarios more complex mappings may of course be used.

As mentioned, the control data may provide mandatory or voluntarycontrol data. Indeed, the received data may include one or more fieldsthat indicate whether the tone mapping parameters provided aremandatory, allowed, or suggested.

In some systems, the display may be capable of operating in accordancewith different dynamic ranges. For example, a very bright HDR displaywith a white point luminance of, say, 5000 nits may also be able tooperate in a display mode with a white point luminance of 4000 nits,another one with 3000 nits, one with 2000 nits, a further with 1000 nitsand finally may operate in an LDR mode having a white luminance of only500 nits.

In such a scenario, the data signal from the display may indicate aplurality of luminance dynamic ranges. Thus, each of the differentluminance dynamic ranges can correspond to a dynamic range mode for thedisplay. In such an arrangement, the dynamic range processor 203 canselect one of the luminance dynamic ranges and proceed to perform thedynamic range transform in response to the selected display dynamicrange. For example, the dynamic range processor 203 may select thedynamic range of 2000 nits and then proceed to perform the dynamic rangetransform to optimize the generated image for this white pointluminance.

The selection of a suitable luminance dynamic range for the display maybe dependent on different aspects. In some systems, the image processingdevice 103 may be arranged to select a suitable display dynamic rangebased on the image type. For example, each range may be associated witha given image type, and the image processing device 103 may select theimage type that corresponds most closely to the received image, and thenproceed to use the dynamic range associated with this image type.

For example, a number of image types may be defined corresponding todifferent content types. For example, one image type may be associatedwith cartoons, another with a football match, another with a newsprogram, another with a film etc. The image processing device 103 maythen determine the appropriate type for the received image (e.g. basedon explicit metadata or on a content analysis) and proceed to apply thecorresponding dynamic range. This may for example result in cartoonsbeing presented very vividly and with high contrast and high brightness,while at the same time allowing e.g. dark films to not be renderedunnaturally.

The system may thus adapt to the specific signals being rendered. Forexample, a poorly made consumer video, a brightly lit football match, awell-lit news program (e.g. scenes with reduced contrast) etc can bedisplayed differently and specifically the dynamic range of the renderedimage may be adapted to that specifically suitable for the specificimage.

It was previously mentioned that the display may provide control data tothe image processing device 103. However, in some systems it mayalternatively or additionally be the image processing device 103 whichprovides control data to the display 107.

Thus, as illustrated in FIG. 22, the image processing device 103 maycomprise a controller 2201 which is capable of outputting a displaycontrol data signal to the display 107.

The display control signal can specifically instruct the display tooperate in the specific dynamic range mode that was selected by theimage processing device 103 for the specific image. Thus, as a result, apoorly lit amateur image will be rendered with a low dynamic rangethereby avoiding introduction of unacceptable errors due to thetransformation to a high dynamic range which is not actually present inthe original image. At the same time, the system can automatically adaptsuch that high quality images can effectively be transformed into highdynamic range images and be presented as such. As a specific example,for an amateur video sequence, the image processing device 103 anddisplay can automatically adapt in order to present the video with a1000 nits dynamic range. However, for a professionally captured highquality image, the image processing device 103 and the display 107 canautomatically adapt to present the video using the full 5000 nitsdynamic range that the display 107 is capable of.

The display control signal may thus be generated to include commandssuch as “use 1000 nits dynamic range”, “use LDR range”, “use maximumdynamic range” etc.

The display control data may be used to provide a number of commands inthe forward direction (from image processing device 103 to display). Forexample, the control data can include image processing instructions forthe display, and specifically can include tone mapping indications forthe display.

For example, the control data may specify a brightness setting, clippingsetting, or contrast setting that should be applied by the display 107.The image processing instruction may thus define a mandatory, voluntaryor suggested operation that should be performed by the display 107 onthe received display signal. This control data can thus allow the imageprocessing device 103 to control some of the processing being performedby the display 107.

The control data may for example specify that a specific filteringshould be applied or should not be applied. As another example, thecontrol data may specify how backlight operations should be performed.For example, the display may be able to operate in a low power modewhich uses aggressive local dimming of a backlight or may be able tooperate in a high power mode where local dimming is only used when itcan improve the rendering of dark areas. The control data can be used toswitch the display between these modes of operation.

The control data may in some examples specify a specific tone mappingthat should be performed by the display, or may indeed specify that tonemapping functions should be switched off (thereby allowing the imageprocessing device 103 to fully control the overall tone mapping).

It will be appreciated that in some embodiments, the system may usecontrol data in both directions, i.e. both in a forwards direction fromthe image processing device 103 to the display 107 and in a backwardsdirection from the display 107 to the image processing device 103. Insuch cases, it may be necessary to introduce operating conditions andrules that resolve potential conflicts. For example, it may be arrangedthat the image processing device 103 is the master device which controlsthe display 107 and overrules the display 107 in case of conflicts. Asanother example, control data may be restricted to specific parametersin the two directions such that conflicts do not occur.

As another example, the master and slave relationships may be usersettable. For example, an image processing device 103 and a display 107may both be arranged to provide control data for the other entity, andmay specifically both be capable of operating as the master device. Theuser may in such systems designate one of the devices to be the masterdevice with the other one becoming a slave device. The user mayspecifically select this based on a preference for him to control thesystem from the image processing device 103 or from the display 107.

The system described above may thus allow communication between contentprovider and image processing device and/or communication between imageprocessing device and display. These approaches could be applied in manysystems that feature a communication channel between a content providerand an image processing device and/or between an image processing deviceand a display. Examples include BDROM, ATSC and DVB, or internet, etc.

The system may utilize a communication channel between an imageprocessing device and a display such as an HDMI or Display portcommunication interface. This communication may be in two directions.E.g., if a smart display is doing all the optimal video and graphicsmapping, the image processing device may e.g. read the controlparameters, and reformat and transmit them in a similar HDMI structure.

The approach may particularly be applied in a BDROM system. As such theapproach may augment BDROM specifications to allow for transmission oftarget display parameters and control commands. Using such data, incombination with end-user display parameters, may allow the BDROM playerto e.g.:

-   -   perform additional video and/or graphics tone mapping or other        processing in the player depending on the characteristics of the        target display and the end-user display.    -   perform additional video and/or graphics tone mapping or other        processing steered by commands in the data stream provided by        the content provider.

In some embodiments, the image processing device 103 may also comprise atransmitter for transmitting dynamic range control data to the contentprovider apparatus 101. Thus, the image processing device 103 may beable to control or at least influence the processing or operationperformed at the content provider apparatus 101.

As a specific example, the control data may include an indication of apreferred dynamic range for the image, and may specifically include anindication of a dynamic range (e.g. white point luminance and optionallyEOTF or gamma function) for the end-user display.

In some embodiments, the content provider apparatus 101 may be arrangedto take the indication of the preferred dynamic range into account whenperforming a tone mapping. However, in other embodiments, the contentprovider apparatus 101 may provide a number of predetermined tonemappings, for example involving a manual tone mapping by a tone mappingexpert. For example, a tone mapped image may be generated for a 500 nitsdisplay, for a 1000 nits display, and for a 2000 nits display.

In such a scenario, the content provider apparatus 101 may be arrangedto select which image to transmit to the image processing device 103based on the received control data. Specifically, the image which isclosest to the dynamic range indicated by the control data may beselected and transmitted to the image processing device 103.

Such an approach may be particularly suitable for a streamingapplication where the streamed signal can dynamically be updated to asfar as possible match the dynamic range of the end-user display.

The approach may reduce the degree of dynamic range transformation thatmust be applied in the image processing device 103 and may specificallyfor scenarios where the content provider apparatus 101 can provide animage tone mapped to the same dynamic range as the end-user displayallow the dynamic range transform to be a simple null operation (i.e. itmay allow the received image to be used directly by the image processingdevice 103.

There are various application scenarios in which the present embodimentscan be useful. E.g., encoding of a particular white point, or intendedwhite, or similar value with the pixel image content (e.g. a DCTencoding of the local object textures), allows for a more smartallocation of the needed code levels versus intended output luminancesfor various possible output signals. One may e.g. encode the texture ofa dark room as if it were well illuminated (i.e. up to pixel lumas of255, rather than having a maximal luma of e.g. 40 in the dark sceneimage), but specify that the “white”, i.e. the 255 has to be treated ina particular way, i.e. that it has to be rendered dark. A simple way todo this is to co-encode e.g. a to be rendered output luminace on thedisplay, for this 255 luma code. The same can be done to encodepredominantly very bright values, such as e.g. in a misty scene withstrong lights in it.

As to the gamma, this can be used e.g. to indicate whether the materialwas encoded e.g. based from gradual celluloid negative material, or witha digital camera with a strong saturation setting. Or any other reasonto deviate from one gamma intention, to another, typically for the finaldisplay on which rendering will occur. EOTFs can typically e.g. encoderather rough grey value behavior, such as compensating e.g. for adisplay with a different gamma, or different viewing environments,compensatable as different gammas. One can hence convey information suchas “signal encoded/intended for, e.g. optimized on, reference display ofgamma=X”, so that another display with other characteristics knows howto process it to get a more optimal rendering towards artist intentions.Tone mappings can be more generic, in that they can also convey e.g.what typical rendering intents were applied to the image (e.g. theartists made the clouds more menacingly dark, which should with anyfinal display rendering mathematics, still show at least approximatelyin the output displayed image).

We elucidate one further example by means of FIG. 23, namely theprinciple of encoding any HDR scene (approximately) in an LDR image(“HDR_encoded_as_LDR”), which could e.g. be a 10 bit image standard, butwe will explain the interesting variant of encoding in a classical 8 bitimage, i.e. an image which is compatible with e.g. an MPEG2 or AVCstandard, and could so be directly used by a classical renderingtechnology. Although one may desire a lot of bits for an HDR signal,e.g. 12, 16 or 22, 8 bits for the luma channel already conveys a lot ofinformation (many possible colors, especially for approximating complextextures) for any peak white of a rendering. Also, many HDR signals mayallow for a significant degree of approximation, since e.g. the sun neednot be encoded exactly with the brightness it really has, since it willbe approximated when rendered on a display anyway. For LDR ranges ofluminance, even a lesser amount of bits will often reasonably suffice,since e.g. 6 bits gives a reasonable approximation/quality of an image(as known from printing).

In the example we hence encode an HDR image exactly within an 8-bit lumastructure, by applying the appropriate mappings i.e. mathematicaltransformations on at least the lumas of the pixels, which are typicallysimple. The criteria are that on the one hand (by co-encoding thetransformations), one can reconstruct the HDR image (i.e. e.g. an 8 bitor 12 bit interpolating approximation intended for a 0.1-5000 nitdisplay rendering) from the LDR 8 bit coded image, by reversing theco-encoded mappings (without the need of any, or significant postcorrection), i.e. the HDR image will look either psychovisually (nearly)indistinguishable, or at least it will still be a good HDR image (i.e.typically show the HDR scene look, approximating how the HDR would berendered if it was generated directly from the original e.g. 12 bit HDRimage IM_HDR, with its HDR range HDR_Rng of to be rendered luminaces).But on the other hand, we desire an LDR image, i.e. if the 8 bit signalwere directly applied to an LDR display of e.g. 0.1-400 nit , whichstill allows for a good visual rendering. E.g., one might just linearlycompress the HDR image IM_HDR to the LDR range LDR_Rng, e.g. by droppingthe least significant bits, and assuming the white (maximum code value255) is intended to be rendered at 400 nit. However, because such HDRimages typically contain very bright objects in the upper part of theirluma range, such an 8 bit image will look too dark on a LDR display,because the relevant darker parts of the image/scene will now end up atvery low luma codes i.e. display output luminances. However, a lot ofimprovement can already be achieved by applying an optimal gamma priorto the encoding of the HDR/12 bit/5000 nit image into the LDR/8 bit/400nit classical e.g. AVC representation. I.e., this gamma will map thebright objects to the brighter parts (e.g. making them less contrastyand pastellish but still acceptable on the LDR display, yet with enoughinformation to do a reasonable reverse mapping to HDR again), optimallycoordinated by at the same time not squeezing the darker parts (e.g.dark tree) too much, so that these dark objects still look reasonablybright on the LDR display (and also a good HDR dark part can berecreated for dark viewing surround viewing; or enough texture data isavailable for brighter encoding of these on the HDR display).

In general such a mapping may be a generic global transformation on thelumas (i.e. a mapping that doesn't take into account geometrical localspecifics, such as where a pixel resides in the image, or what the lumasof its neighboring pixels are, or what kind of scene object it belongsto, but rather only takes as input the luma value of the pixel).Somewhat more complex mappings may be co-encoded, such as atransformation only for a demarcated subregion or object in the image(local mapping, in which case typically further information isco-encoded such as defining the boundary of the object). But in general,although one could envisage any transformation to work with ourdisclosed embodiments, be it only to reduce the amount of work oftypically a human grader defining these optimal mappings, they willtypically be few and simple (no local mapping will be encoded if ageneral global function such as an S-curve or multipoint splinesuffices).

We elucidate the example with a content creator side image encodingapparatus 510, with human optimized encoding of the output image beingtypically an 8 bit LDR image Im_1 (as typically encompassed with thetransformation/mapping functions or algorithmic strategies as metadataMET in some image signal structure S such as prescribed in AVC or HEVC)to a memory (such as a blu-ray disk 511, or a temporary memory, forultimate encoding on a signal to be stored or transmitted). This gradermay check the image on one or more displays 530, e.g. checking whetherboth the LDR and recoverable HDR image look alright on respectivereference LDR and HDR displays, before sending his instructions to theimage encoding unit 550 (which does the mapping to the 8 bit luma) andthe formatter 554, which finalizes the image and its color codesaccording to the currently used image coding standard, and co-encodesthe texture image with the metadata of the transformation to an output512.

In the top part we see how the HDR image IM_HDR (which is inputted viaan input 511 of the image encoding apparatus 510) with its HDR range ismapped to the LDR image with its LDR range of rendered luminances if onan LDR display.

Although we elucidated the “HDR_encoded_as_LDR” with an encoding on acontent creation side for transmission to a content usage side such as aconsumer's home, the same “HDR_encoded_as_LDR” embodiments can obviouslyalso be used when transmitting (e.g. by transcoding) between differentapparatuses, such as e.g. two home apparatuses in a home network. Thene.g. an automatic image analysis and mapping unit may apply an automaticimage analysis and a corresponding luma mapping method. This can be donee.g. by a content receiving or storing apparatus when having a firstimage representation, such as e.g. the 12 bit HDR image, and sending itover a HDMI or other network connection, to a television. Or the 8 bitLDR image may be encoded according to or for a wireless standard, forstreaming to a mobile display, with HDR capabilities, yet of lesservisual quality anyway.

Typically, at least for new HDR standards, in case such a 8 bit encoding(e.g. 8 bit luma and normal 2×8 bit encodings for chroma) in a classicalLDR (e.g. MPEG) scheme is done, the standard will annotate in metadatathat this LDR image is actually not an LDR image primarily intended forLDR displays (although as said, it may have been constructed so that itstill looks reasonable on an LDR display of e.g. 100 nit peak brightnessor peak white), but is an HDR image. It may do so with a generic HDRcode, which is e.g. assumed to give reasonable renderings for HDRdisplays with peak brightness around 3500 nit. The in metadataco-encoded first target display reference can also be somewhat morespecific, in that this HDR signal was originally graded on e.g. a 5000nit display. This will mean that the actual lumas of the image objects(also when mapped into LDR lumas) will have values depending on what istypically rendered on a 5000 nit display (e.g. reserving a subrange ofhigh brightness, and pushing normal brightness scene object, typicallythe main objects in the scene, towards deeper luma values, already inthe e.g. 16 bit HDR raw grading). In this case an actual 3500 nit or2500 nit display, rather than to just use the HDR signal for driving thedisplay assuming it will still give a reasonable picture (the peakbrightness being in a range around the intended value 5000 nit), canfurther optimize is color transformation functions for optimally gamutmapping to the actual display gamut according to a quality criterion(e.g. output luminance similarity, or a psychovisual appearance qualitymeasure on the HDR effects, etc.). One could even co-encode a secondpeak brightness value for the final HDR_encoded_as_LDR image (e.g.,looks most reasonable on a 250 nit display, starting to show slightartefacts on higher and or lower peak brightnesses, potentially evenfurther specifying such artefacts, preferably in a functional way (e.g.geometrical location, etc.), so that a renderer can attempt to correctfor the artefacts).

With HDR display we mean a display of peak brightness greater than 750nit, displays with lower peak brightness, and especially below 500 nitbeing LDR displays.

The predetermined quality criterion for judging whether the LDRrendering, and the HDR rendering of a recovered HDR signal from the LDRimage (typically derived solely by inverting the co-encoded mappings,but some further processing may be done, like the receiving sideapparatus may apply a quantization boundary mitigating image processinge.g.), will be either a mathematical algorithm, or the human operatorjudging that it is good enough when encoding the final image codings fordistribution. Both the human applied and software encoded qualityestimators will apply such image analysis criteria as: is theresufficient (local) contrast in various regions (i.e. still retainingenough of the visibility of the original e.g. master celluloid negativescan 12 or 14 bit HDR image), in particular the regions central in theimage, are there many artefacts like quantization boundaries, and howlarge or wide are the steps, are there sufficient spatial submodes ofthe luminance histogram (is the original cinematic look/intentretained), in particular, have spatially separated objects sufficientinter-region contrast, etc.. And in particular, if originals arepresent, like e.g. in a networked system of connected apparatuses, thesending apparatus (e.g. a settopbox) judging whether the recoverable HDRsignal is sufficiently close to the original e.g. 12 bit HDR signalpresent at that location (which may be done based on such mathematicalcriteria like MSE or PSNR, or psychovisually weighed differences, etc.).E.g., after an automatic luma transformation, and correspondingautomatic color adjustment (which may e.g. be a gamma function orsimilar power function, or an S-curve, tuned on such factors like atypical e.g. median brightness in the scene, or further image analysislike detection of small bright regions and giving them their ownsubrange and corresponding mapping function, etc.), a color grader (e.g.after first having done the master grading on the 16 bit original HDR)will then further color grade the HDR_encoded_as_LDR image. On the onehand this is hence done to give a nice usable LDR grading, but on theother hand also a recoverable HDR, so he may allocate importantinformation containing regions to subregions of the LDR range which haveenough code values, but still shift them to “average” luma ranges whichshow a good rendering on LDR (e.g. not too dark, so that the darkerregions are still well visible, yet dark enough to still convey themood). Typically he may do so by tweaking the luma/color mappingfunction(s) from the automatic ones. At least the lumas should becorrectly positioned, the color can then be optimized starting fromthat. E.g., if a certain background region came out to darkish in theLDR rendering, he may still tune the global mapping function in the partcorresponding to those pixels, provided it doesn't become worse in otherparts of the LDR rendering, and of course via inverse mapping therecoverable HDR image doesn't become of subcritical quality. Inprinciple he could even choose to grade a spatially local image region(be it for the first time, or a second time in addition to a first imageencoding for that region) so that it corresponds to another display peakbrightness, or gamma etc., than the rest of the image, so that prior torendering the renderer would have to take that into account. This couldbe useful e.g. to emphasize dark regions, but in general one would keepthings simply fixed to one HDR intended display encoding. So then alsothe global mapping function from HDR-to-LDR (or its inverse, mappingLDR-to_HDR) and if applicable further transformation data is co-encoded.Legacy systems will ignore all of that, and in principle can use theclassically coded LDR image even if the first target display referenceand other information is dropped from the image coding signal, but ingeneral one will write this in sectors of the data which were e.g.reserved for upgrading, and are ignored by older systems, but used bythe newer ones. A HDR image decoding box could take a look at this dataanyway, even though it may be connected to an old LDR display. Insteadof just applying the LDR signal to the display for driving therendering, it could then improve the

LDR signal somewhat by a color transformation, given all local factors,and all this additional metadata information (whether using just thetype of display that was intended, and “blindly” transforming the LDRinputed signal based on similarity of the current rendering environmentwith the intended one the grader was working with, or by also using someor all of the information in the mapping functions between the LDRencoding and the original HDR encoding, which information says somethingabout the difference between the two, i.e. the HDR nature andcomposition of the original scene and/or graded HDR signal) .

Such a signal has the advantage that any HDR-capable system knows thatwe have actually an HDR image encoded as an LDR one, and can optimallyrecover that HDR image before rendering, yet backwards-compatible,legacy LDR systems can also directly use the LDR image for rendering.

It should be clear to the skilled persons which combinations can be madefrom our teachings, such as e.g. encoding several HDR gradings forseveral HDR displays, e.g. in several LDR encodings, regradings,alternatives for different situations such as a change in viewingenvironment which can also be seen as a display rendering type. Where wemention specific parameters such as 8 bit legacy encoding, of course itshould be understood the same can be done with e.g. a 10 bit LDRencoding technology, and we don't want to exclude from protection anyvariant, combination, or simple alternative realization. So theteachings of our claims can of course be combined, and are considered sodescribed without the tediousness of explicit specification of each andevery easily co-readable variant, unless it is clear especially from ourdescription that such isn't possible or intended. Of course theencodings can be used in various scenarios, whether professional or e.g.mobile consumer, several applications like e.g. security systems, newsgathering, etc. It can be used largely automatically inside anytechnical system, like within an IC or multichip, or networked technicalsystem, etc. Some of the parts of the invention may form separatebusiness applications, like e.g. any of the gradings can be performed asa regrading on an existing image encoding, whether already graded inthis way, but now improved, or lacking some kind of grading, like for anovel popular display or way of dipslaying.

It will be appreciated that the above description for clarity hasdescribed embodiments of the invention with reference to differentfunctional circuits, units and processors. However, it will be apparentthat any suitable distribution of functionality between differentfunctional circuits, units or processors may be used without detractingfrom the invention. For example, functionality illustrated to beperformed by separate processors or controllers may be performed by thesame processor or controllers. Hence, references to specific functionalunits or circuits are only to be seen as references to suitable meansfor providing the described functionality rather than indicative of astrict logical or physical structure or organization.

All method embodiments and teachings correspond to correspondingapparatus, and potentially further product such as output signals,embodiments, and vice versa. The invention can be implemented in anysuitable form including hardware, software, firmware or any combinationof these. The invention may optionally be implemented at least partly ascomputer software running on one or more data processors and/or digitalsignal processors. The elements and components of an embodiment of theinvention may be physically, functionally and logically implemented inany suitable way. Indeed the functionality may be implemented in asingle unit, in a plurality of units or as part of other functionalunits. As such, the invention may be implemented in a single unit or maybe physically and functionally distributed between different units,circuits and processors.

Although the present invention has been described in connection withsome embodiments, it is not intended to be limited to the specific formset forth herein. Rather, the scope of the present invention is limitedonly by the accompanying claims. Additionally, although a feature mayappear to be described in connection with particular embodiments, oneskilled in the art would recognize that various features of thedescribed embodiments may be combined in accordance with the invention.In the claims, the term comprising does not exclude the presence ofother elements or steps.

Furthermore, although individually listed, a plurality of means,elements, circuits or method steps may be implemented by e.g. a singlecircuit, unit or processor. Additionally, although individual featuresmay be included in different claims, these may possibly beadvantageously combined, and the inclusion in different claims does notimply that a combination of features is not feasible and/oradvantageous. Also the inclusion of a feature in one category of claimsdoes not imply a limitation to this category but rather indicates thatthe feature is equally applicable to other claim categories asappropriate. Furthermore, the order of features in the claims do notimply any specific order in which the features must be worked and inparticular the order of individual steps in a method claim does notimply that the steps must be performed in this order. Rather, the stepsmay be performed in any suitable order. In addition, singular referencesdo not exclude a plurality. Thus references to “a”, “an”, “first”,“second” etc do not preclude a plurality. Reference signs in the claimsare provided merely as a clarifying example shall not be construed aslimiting the scope of the claims in any way.

1. An image processing apparatus comprising: a receiver for receiving animage signal, the image signal comprising at least a first encoded imageand in addition a first target display reference, the first targetdisplay reference being indicative of a dynamic range of a first targetdisplay for which the first encoded image is encoded, and the firsttarget display reference comprising an Electro Optical Transfer Functionindication for the first target display: a dynamic range processorarranged to generate an output image by applying a dynamic rangetransform to the first encoded image by using the Electro OpticalTransfer Function indication for the first target display; and an outputfor outputting an output image signal comprising the output image. 2.The image processing apparatus of claim 1 wherein the first targetdisplay reference comprises a tone mapping indication representing atone mapping used to generate the first encoded image for the firsttarget display.
 3. The image processing apparatus of claim 1, whereinthe image signal further comprises a data field comprising dynamic rangetransform control data; and wherein the dynamic range processor isfurther arranged to perform the dynamic range transform in response tothe dynamic range transform control data.
 4. The image processingapparatus of claim 1, wherein the dynamic range processor is arranged todetect the first encoded image being an LDR image encoded as a legacyLDR encoding such as an MPEG-AVC encoding, is arranged to read frommetadata associated with the image signal a dynamic range transformcorresponding to the first target display reference, and is arranged togenerate the output image being a high dynamic range image graded forrendering on the first target display by applying the dynamic rangetransform to the first encoded image.
 5. The image processing apparatusof claim 3 wherein the dynamic range transform control data comprisesdifferent dynamic range transform parameters for different displaymaximum luminance levels.
 6. The image processing apparatus of claim 3,wherein the dynamic range transform control data comprises differenttone mapping parameters for different display maximum luminance levels,and wherein the dynamic range processor is arranged to determine tonemapping parameters for the dynamic range transform in response to thedifferent tone mapping parameters and a maximum luminance for the outputimage signal.
 7. The image processing apparatus of claim 3, wherein thedynamic range transform control data comprises data defining a set oftransform parameters that must be applied by the dynamic rangetransform.
 8. The image processing apparatus of claim 3, wherein thedynamic range transform control data comprises data defining limits fortransform parameters to be applied by the dynamic range transform. 9.The image processing apparatus of claim 3, wherein the dynamic rangetransform control data comprises different transform control data fordifferent image categories.
 10. The image processing apparatus of claim1 wherein a maximum luminance of the dynamic range of the first targetdisplay is no less than 1000 Cd/m̂2.
 11. The image processing apparatusof claim 1 further comprising: a receiver for receiving a data signalfrom a display, the data signal comprising a data field which comprisesa display dynamic range indication of the display, the display dynamicrange indication comprising at least one luminance specification; andwherein the dynamic range processor is arranged to apply the dynamicrange transform to the first encoded image in response to the displaydynamic range indication.
 12. The image processing apparatus of claim 1wherein the dynamic range processor is arranged to select betweengenerating the output image as the first encoded image and generatingthe output image as a transformed image of the first encoded image inresponse to the first target display reference.
 13. The image processingapparatus of claim 1 wherein the dynamic range transform comprises acolor gamut transform.
 14. The image processing apparatus of claim 1further comprising a control data transmitter for transmitting dynamicrange control data to a source of the image signal.
 15. An image signalencoding apparatus comprising: a receiver for receiving an encodedimage; a generator for generating an image signal comprising the encodedimage and a target display reference indicative of a dynamic range of atarget display for which the encoded image is encoded, which targetdisplay reference comprises an Electro Optical Transfer Functionindication for the target display; and a transmitter for transmittingthe image signal, comprising the encoded image and the target displayreference.
 16. The image signal encoding apparatus of claim 15 whereinthe target display reference comprises a tone mapping indicationrepresenting a tone mapping used to generate the first encoded image forthe first target display.
 17. The image signal encoding apparatus ofclaim 15 wherein the generator is further arranged to generate the imagesignal to comprise a data field including dynamic range transformcontrol data; the dynamic range transform control data being indicativeof a parameter of a dynamic range transform for the encoded image. 18.An image processing method comprising: receiving an image signal, theimage signal comprising at least a first encoded image and a firsttarget display reference, the first target display reference beingindicative of a dynamic range of a first target display for which thefirst encoded image is encoded, and the first target display referencecomprising an Electro Optical Transfer Function indication; generatingan output image by applying a dynamic range transform to the firstencoded image in response to the Electro Optical Transfer Functionindication; and outputting an output image signal comprising the outputimage.
 19. A method of transmitting an image signal which encodes pixelluminances which can have values up to a maximum being a least 1000Cd/m̂2, the method comprising: receiving an encoded image; generating animage signal comprising the encoded image and in addition a targetdisplay reference indicative of a dynamic range of a target display forwhich the encoded image is encoded, the target display referencecomprising an Electro Optical Transfer Function indication; andtransmitting the image signal.